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ABSTRACT 

This  study  established  the  feasibility  of  using  information 
measurement  techniques  to  analyze  reported  maintenance  data. 
Failure  data  on  two  systems  were  analyzed;  1)  the  ASB-4  Bomb/ 
Navigation  System  of  the  B-52,  and  2)  the  MA-1  Fire  Control 
System  of  the  F-I06.  The  ASB-4  data  (flight-line  reports)  and 
the  shop  data  for  the  MA-1  contained  written  descriptions  of  the 
malfunction  sjnnptoms.  To  make  these  data  eimenable  to  multi- 
variate uncertainty  analysis,  a coding  scheme  designed  to  retain 
the  grammatical  form  and  content  of  these  symptom  reports  was 
used.  The  F-106  organizational  (flight-line)  data  are  already 
coded  (according  to  ADC  66-28  symptom  codes)  which  make  them 
directly  amenable  to  information  analysis. 

In  analyzing  the  ASB-4  data,  it  was  found  that  the  subject 

of  the  reported  symptom  was  most  indicative  of  the  unit  contain- 

ing the  malfunction,  while  the  predicate  phrases  and  modifiers 
showed  a relationship  with  repair  time.  The  F-IO6  data  Indicated 
a strong  relation  between  pilot  and  symptom,  although  the  nature 
of  this  relationship  cannot  be  determined  from  the  available  data 

Some  pilots  may  not  be  reporting  symptoms  when  they  should 

be . Introducing  a training  program  on  Symptom  Reporting  Proce- 

dures should  markedly  Increase  systems  effectiveness . It  appears 
that  a change  in  Job  assignment  and  personnel  evaluation  proce- 
dures could  have  a large  Impact  on  maintenance  workload.  Addi- 
tional data  are  required  before  specific  recommendations  can  be 
made,  but  the  results  of  this  study  are  highly  suggestive  of  the 
type  of  data  needed  to  ajiswer  the  questions  raised. 
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SECTION  I 


INTRODUCTION 

When  operators  complain  about  equipment  performance,  they 
are  trying  to  tell  maintenance  men  something  about  how  the 
equipment  Is  functioning;  they  may  also,  albeit  unwittingly, 
tell  analysts  of  maintenance  data  something  about  how  they,  the 
operators,  are  performing.  Simultaneously,  when  maintenance 
personnel  perform  maintenance  actions,  the  equipment  "tells" 
them  how  Is  performing,  and  the  reported  actions  tell  ana- 
lysts of  maintenance  data  something  about  how  the  maintenance 
men  are  performing. 

Figure  1 Is  a symbolic  representation  of  this  communication 
of  Information,  as  It  Is  reviewed  by  an  analyst  of  maintenance 
data.  In  Path  1,  equipment  malfunctions  are  transformed  Into 
S3nnptoms;  In  Path  2,  symptoms  are  observed  and  reported  to  main- 
tenance men  by  operators;  In  Path  3>  maintenance  men  use  test 
equipment,  ground  power  sources,  etc.,  to  diagnose  the  equipment; 
In  Path  4,  maintenance  men  fill  out  reports  telling  analysts 
what  maintenance  was  required;  Path  5 Indicates  reported  malfunc- 
tions are  Influenced  by  external  pressures  applied  to  operators 
(e.g.,  evaluation  requirements,  training  material,  personal 
biases);  and  Path  6 shows  that  reported  maintenance  actions  are 
Influenced  by  external  pressures  applied  to  maintenance  personnel 
(e.g.,  fear  of  repeat  write-up  "black  marks",  training  material, 
personal  biases). 

It  Is  Important  to  understand  how  Information  Is  transmitted 
In  this  system  and  how  well  It  Is  transmitted.  For  If  this  Infor- 
mation Is  not  being  transmitted  efficiently  — and  there  Is  con- 
siderable evidence  that  It  Is  not  — then  mission  performance 
probably  can  be  Improved,  perhaps  dramatically,  by  simply  Improv- 
ing the  transmission  efficiency  of  maintenance  Information,  thus 
Improving  the  efficiency  of  maintenance  Itself. 

This  study  Is  a "first  examination"  of  this  problem  from  a 
systems  point  of  view,  using  Information  theory  as  an  analytic 
tool.  This  was  a small  study  (one  man-year),  using  data  collected 
for  other  purposes  and  applying  certain  concepts  for  the  first 
time.  As  a consequence  of  these  limitations,  several  conclusions 
may  be  tenuous . Even  so,  the  study  does  demonstrate  that  Infor- 
mation theory  can  be  used  as  aji  analytic  tool  for  measuring  cer- 
tain maintenance  characteristics.  The  results  presented  In  this 
report  are  based  upon  analyses  of  data  collected  for  earlier 
reliability  and  maintainability  studies,  and  not  upon  data  col- 
lected specifically  for  the  purpose  of  Information  analysis. 
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BACKGROUND 


1 . 

For  several  years,  many  agencies  and  Investigators  have 
studied  facets  of  the  problems  displayed  In  Figure  1.  At  ARINC 
Research  Corporation,  where  the  present  research  effort  was 
performed,  these  studies  were  primarily  directed  at  Paths  1,  3> 
and  4.  Concern  was  with  how  equipment  malfunctions  are  displayed 
to  operators  (Path  l),  with  the  effectiveness  of  ground  test 
equipment  (Path  3),  and  with  the  reliability  and  maintainability 
of  operational  systems  (Path  4) . 

For  the  first  path,  a theoretical  model  was  developed  which 
predicted  the  displayed  symptoms  as  a result  of  specified  degra- 
dation of  equipment  components  (ref.  l).  The  procedures  developed 
allowed  for  the  preparation  of  symptom/cause  tables  (thus  the 
name  "The  Symptom  Matrix")  like  that  Illustrated  In  Table  1. 

These  tables  can  be  developed  either  (a)  from  early  design  stage 
engineering  Information,  or  (b)  from  operational  failure  and 
repair  Information.  The  tables  can  be  used  to  determine  the 
need  for  Improved  troubleshooting  tools,  strategies,  and  documen- 
tation and  assist  In  the  maintenance  and  Improvement  of  required 
personnel  skills.  If  the  symptom  matrix  Is  produced  during  sys- 
tem design.  It  can  be  used  as  a design  tool  for  making  recommen- 
dations concerning  packaging,  the  location  and  number  of  test 
points,  and  the  type  and  output  of  "front-panel  Indicators."  At 
the  same  time  It  can  be  used  to  anticipate  tools,  spares,  train- 
ing, and  training  aid  requirements . 

For  the  third  path,  attention  was  directed  toward  (a)  the 
adequacy  of  ground  test  equipment  to  verify  "in-f light"  symptoms, 
(b)  the  ability  of  test  equipment  In  general  to  reject  "baid"  com- 
ponents and  accept  "good"  components,  and  (c)  the  cost  effective- 
ness of  automatic  test  equipment.  These  studies  were  always 
hardware  oriented  and  always  system  specific. 

Attention  given  to  the  fourth  path  dealt  with  how  much  con- 
fidence maintenance  data  analysts  could  place  on  reported  mainte- 
nance data.  If  these  data  are  to  be  used  as  an  informatJon  source 
for  reliability  and  maintainability  evaluation  and  improvement 
studies,  they  must  be  reasonably  accurate  and  relatively  complete. 
Independent  checking  of  AFM  66-1*  data  led  to  the  conclusion  that 
for  purposes  of  detailed  evaluation  Information,  these  data  were 
not  adequate  (ref.  2).  Coverage  was  good,  but  details  were  miss- 
ing and  accuracy  was  low. 

Although  these  studies  flirted  with  it,  none  directly 
attacked  the  problem  of  personal  bias  In  data  reporting  or  per- 
sonnel effects  upon  system  effectiveness.  If  Channel  1 Is 


* "Maintenance  Management:  Organizational,  Field  and  Depot", 
15  June  1966. 
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operating  perfectly,  l.e..  If  the  system  Is  designed  so  that  It 
unambiguously  displays  symptoms  so  that  each  distinct  symptom  Is 
related  to  exactly  one  discrete  maintenance  action,  then  It  Is 
still  possible  for  the  operator  (Channel  2)  to  misread  or  mls- 
report  symptoms  and  to  mislead  the  maintenance  man.  (If  the  test 
equipment  Is  effective,  Channel  3>  the  misleading  nature  of  this 
Information  can  be  determined  quickly  and  unequivocally.  If  the 
maintenance  man  does  the  right  things,  then  on  the  next  mission 
the  equipment  should  perform  properly  and  the  maintenance  man's 
report  (Channel  4)  should  accurately  Indicate  what  malfunction 
had  occurred. 

Careful,  cyclical  review  of  the  channels  of  Information  flow 
shown  In  Figure  1 will  Indicate  that  If  the  test  equipment  In  the 
system  (Channel  3)  performs  satisfactorily,  then  the  operator  Is 
more  or  less  irrelevant  from  a maintenance  point  of  view.  How- 
ever, reflection  on  the  status  of  current  and  proposed  test  equip- 
ment designs  Indicates  (a)  that  satisfactory  test  equipment  diag- 
nosis, Independent  of  mission  performance  Information,  is  not  to 
be  expected,  and  (b)  even  perfect  test  equipment  Is  a poor  alter- 
native to  adequate  failure  reporting  — why  waste  time  replicating 
adequate  operator  malfunction  reports?  The  problem  Is  one  of 
finding  out  who  and  what  to  "trust"  with  respect  to  how  the  system 
Is  or  has  been  performing.  If  this  can  be  accomplished  effectively, 
test  equipment  need  only  be  used  for  those  situations  where  ambi- 
guity remains,  or  — Ideally  — at  the  next  level  of  repair  need- 
ing information  beyond  that  contained  In  the  ssraiptoms . 

2 . PURPOSE 

This  research  was  designed  to  analyze  a representative  sample 
of  Air  Force  failure  reports  and  the  resulting  set  of  maintenance 
actions,  to  Identify  possible  relationships  between  the  informa- 
tion content  of  the  reported  symptoms  In  these  failure  reports 
and  maintenance  efficiency.  In  addition  to  describing  the  above 
relationships,  (the  study  was  Intended  to)  we  hoped  to  establish 
the  feasibility  of  trying  to  improve  maintenance  efficiency 
through  manipulation  of  failure  report  content. 

Initially,  the  effort  was  viewed  as  exploratory  and  was  pri- 
marily directed  toward  the  first  of  the  two  Issues  cited  above 
(identifying  relationships  between  failures  reports  and  mainte- 
nance results).  The  study  would  be  considered  a success  If  It 
objectively  evaluated  the  feasibility  of  Information  theory  as 
a maintenance  data  analysis  tool.  Heretofore,  published  mainte- 
nance analyses  contained  tantalizing  suggestions  that  Information 
theory  could  be  applied  to  maintenance  data,  but  these  studies 
had  not  directly  approached  the  Issue.  Therefore,  the  activities 
Initially  planned  for  this  study  Included,  In  sequence: 

(l)  Given  a representative  sample  of  Air  Force  failure 

reports,  attempt  to  measure  the  amount  of  information 
communicated  to  the  maintenance  man  by  the  equipment 
operator. 


5 


(2)  If  this  Is  successful,  describe  how  the  linguistic 
content  of  these  failure  reports  Is  related  to  the 
maintenance  man's  malfunction  diagnosis. 

(3)  Develop  some  general  conclusions  on  how  to  Improve 
maintenance  efficiency  through  manipulation  of  the 
failure  reporting  process. 

Later,  as  the  results  of  this  Investigation  began  to  take 
form,  the  areas  of  Interest  were  expanded  to  take  advantage  of 
some  additional  data  which  had  become  available.  These  data, 
and  the  analysis  which  had  already  been  accomplished,  made  It 
clear  that  there  was  much  more  to  Information  theory  as  a main- 
tenance analysis  tool  than  had  been  anticipated  and  that  changes 
In  maintenance  reporting  procedures  were  not  the  only  factors 
that  could  appreciably  affect  maintenance  efficiency. 

Accordingly,  the  study  was  expanded  to  Include  the  follow- 
ing additional  activities; 

(4)  Evaluate  the  potential  of  Information  theory  as  a 
general  tool  for  Identifying  the  structure*  of  main- 
tenance data. 

(5)  Develop  some  recommendations  on  how  to  Improve  system 
efficiency  through  manipulation  of  the  total  failure 
reporting  and  maintenance  process. 


* "Structure"  here  Is  a technical  word  meaning,  roughly,  how 
the  elements  being  studied  fit  together  and  how  they  are 
mutually  related  — see  Reference  3. 
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SECTION  II 
DATA 

The  data  used  in  this  study  were  collected  for  two  previous 
amd  unrelated  programs  with  other  purposes  In  mind.  Since  the 
einalyses  (and  their  Interpretations)  depend  upon  the  specific 
nature  of  this  data,  an  understanding  of  its  background  and 
nature  Is  essential  for  an  understanding  of  this  report. 

1.  WALKER  AIR  FORCE  BASE 

Data  collection  for  an  Air  Force  reliability  assessment  and 
Improvement  program  began  In  March  1957  and  continued  until 
July  1963.  Between  March  i960  and  April  I96I,  data  were  collected 
on  all  maintenance  activity  on  the  ASB-4  Bomblng/Navlgatlon  sys- 
tems  'installed  In  one  wing  of  B-52  aircraft  at  Walker  Air  Force 
Base . 

ARINC  Research  field  engineers  obtained  this  data  from 
debriefing  Interviews  with  flight  line  maintenance  personnel 
Immediately  after  the  latter  returned  from  a maintenance  action 
at  the  aircraft.  A complete  Incremented  time  history  of  each 
event  was  recorded,  accounting  for  all  elapsed  time  from  receipt 
of  the  work  order  to  completion  of  repair.  Details  of  actions 
taken,  units  Involved,  and  the  circumstances  encountered  were 
also  recorded.  In  addition,  the  symptoms  which  Initiated  the 
maintenance  actions  were  recorded  verbatim.  In  narrative  form. 

Any  hesitancy  which  the  maintenance  men  may  have  had  toward 
reporting  possible  uneconomical  usage  of  time  was  minimized  (hope- 
fully) by  the  excellent  rapport  established  between  them  and  the 
ARINC  Research  personnel,  and  their  assurance  from  the  base 
commander  that  details  of  the  raw  data  were  available  only  to 
ARINC  Research. 

The  validity  of  the  data  obtained  was  checked  by  Informally 
comparing  It  with  a control  sample  of  data  obtained  by  direct 
observation  of  several  maintenance  events  at  the  aircraft.  No 
significant  difference  was  found  between  the  data  samples  obtained 
using  the  two  collection  methods. 

Appendix  I shows  all  the  variables  considered  In  the  data 
collection  at  Walker  APB  during  the  period  March  i960  to 
April  1961.*  There  were  1722  reports  Involving  the  ASB/4 


*The  data  collected  during  this  period  differ  from  those  of  the 
total  data  collection  period  In  two  respects.  These  cover  all 
maintenance  action  on  the  aircraft  and  systems  under  observation 
and  these  also  include  the  verbatim  malfunction  report  - the 
"symptom. " 


bombing-navigation  system.  These  reports  were  abstracted  and 
used  as  the  primary  data  for  this  study.  A "working"  IBM  card 
format  was  devised  for  these  reports;  the  format  Included  the 
data  elements  listed  in  Table  2. 

TABLE  2 

asb/4  plight  line  data 

USED  IN  THIS  STUDY 


Variable  - Code  Explained  In: 

Appendix 

Number 

Item 

Number 

1.  Aircraft  Identification 

I 

1 

2.  Report  Number 

I 

26 

3.  Grammatically  Coded  Symptom 

TV 

-- 

4.  Number  of  Phrases  In  the  Symptom 

TV 

-- 

5.  Whether  or  not  the  Symptom  was 

Verified 

I 

29 

6.  The  Units  Involved  In  the  Action 

I 

12 

7.  Numbers  and  Skill  Levels  of  Malnte- 

nance  Men  Invc Ived 

I 

30 

8.  Maintenance  Time  Variables 

I 

31,  35  & 40 

For  some  reports.  Information  is  available  concerning  the 
Identity  of  the  personnel  Involved  In  the  maintenance  action,  as 
well  as  personal  Information  (test  scores,  IQs,  etc.)  for  these 
men.  Sample  sizes  and  problems  associated  with  defining  equiva- 
lent tasks  and  skills  made  analysis  of  these  Items  unfruitful. 
Perhaps  with  fuither  attention  they  can  be  led  to  produce  Inter- 
esting results. 

2.  DOVER  AND  SELFRIDGE  AIR  FORCE  BASES 

Data  collection  for  an  Air  Force  reliability  and  maintain- 
ability Improvement  program  for  F-106  electronic  subsystems  was 
begun  In  May  1964  and  Is  still  In  process.  Between  November  1965 
and  February  1966,  data  were  collected  on  all  maintenance  activity 
on  the  avionics  subsystems  Installed  In  the  F-IO6  aircraft  at 
Selfrldge  and  Dover  Air  Force  Bases. 

The  data  were  obtained  from  Air  Force  maintenance  records 
(Forms  AF  76-3^  AFTO  210,  and  APTO  211)  and  were  supplemented 
with  individual  follow-up  by  a field  engineer.  A complete  repair 
history  of  each  event  was  recorded,  accounting  for  the  Identity 
of  the  operational  and  maintenance  personnel,  the  operational 
symptoms  (ADC  66-28  Codes),  and  the  flight  line  diagnosis,  the 
symptom  reported  to  the  shop  (verbatim  report;  the  flight  line 
"How  Mai"  code  was  not  analyzed)  and  the  shop  repair  action. 


and  the  system  performance  on  the  next  mission.  These  data 
Include  1809  flight  line  maintenance  actions  (collected  at  both 
bases)  and  407  shop  maintenance  actions  (collected  at  Dover  AFB 
only) . 

Since  the  data  were  generally  drawn  from  Air  Force  records, 
filled  out  by  Air  Force  maintenance  personnel  as  part  of  their 
dally  routine,  any  errors  or  misinformation  which  might  be  pres- 
ent in  Air  Force  maintenance  paper  work  are  also  reflected  In 
these  data.  As  It  happens.  In  a later  section  of  this  report, 
this  opportunity  for  bias  will  be  shown  to  have  some  alarming 
Implications  with  respect  to  system  performance. 

Appendix  II  shows  all  of  the  variables  which  were  considered 
In  the  data  collection  at  Dover  and  Selfrldge  AFBs.  There  are 
1268  flight  line  reports  from  Selfrldge  and  54l  flight  line 
reports  from  Dover.  All  of  these  reports  concern  malfunctions 
of  the  electronic  subsystems  In  F-IO6  aircraft.  "Working"  IBM 
card  formats  were  devised  for  these  data.  Table  3 presents  the 
F-106  flight  line  maintenance  data  elements  studied  here. 

Table  4 performs  the  same  function  for  the  shop  data. 

3 . SYMPTOMS 

The  subject  of  this  research  Is  symptoms,  and  their  ability 
to  guide  maintenance.  But  since  very  little  Is  known  about  this 
problem,  it  seems  appropriate  to  give  a little  attention  to  at 
least  two  major  Issues;  (l)  how  symptoms  are  reported  and  (2) 
how  reported  symptoms  can  be  coded  for  analysis.  The  details  of 
these  two  Issues  are  given  in  Appendices  III  and  IV,  respectively. 

3.1  Symptom  Reporting 

Symptom  reporting  is  a matter  of  some  concern  because  (a)  an 
objective  of  the  study  Is  to  Improve  the  process,  eind  (b)  the 
primary  ssnnptom  data  were  collected  on  the  ASB/4  nearly  five 
years  before  this  analysis  began.  At  the  outset  of  this  study, 
there  was  some  question,  about  these  data,  still  representing 
Air  Force  maintenance  practice . 

The  detailed  material  In  Appendix  III  Indicates  that  symptom 
reporting  procedures  at  the  various  Air  Force  bases  sampled  are 
similar  In  most  respects,  whether  they  are  SAC  bases  or  TAG  bases, 
and  whether  the  symptom  reporting  process  was  observed  In  1959 
or  1966.  All  of  the  bases  use  the  debriefing  procedure  outlined 
in  AFM  66-1,  and  In  general,  the  "operator  - debrlefer  - mainte- 
nance control  - supervisor  - flight  line  personnel"  chain  men- 
tioned In  AFM  66-1  was  also  used  both  then  and  now.  The  biggest 
consistent  difference  seems  to  be  whether  or  not  flight  line 
maintenance  personnel  routinely  have  direct  (or  semldlrect)  con- 
tact with  system  operators  prior  to  debriefing.  There  are 


9 


TABLE  3 


f-106  plight  line  data 

USED  IN  THIS  STUDY 


Variable  - Code  Explained  In: 

Appendix 

Number 

Item 

Number 

1 . Base 

II 

1 

2.  Aircraft  Identification 

II 

4 

3.  Date  of  Maintenance 

II 

6 

4.  Pilot's  Identification  Code 

II 

3 

5.  Symptom  Code 

II  & III 

16 

6.  Malfunction  Verification 

II 

19 

7.  Maintenance  Supervisor's  Identlflca- 

tlon  Code 

II 

36 

8.  Maintenance  Asslstajit's  Identlflca- 

tlon  Code 

II 

37 

9.  Unit  Involved  In  the  Action 

II 

25 

10.  Action  Taken 

II 

20 

11.  Troubleshooting  Method 

II 

22 

12.  Maintenance  Concept  (Team  Approach) 

II 

21 

13.  Whether  the  Equipment  Operated 

Successfully  on  the  Next  Plight 

II 

23 

l4.  Total  Subsystem  Downtime 

II 

24 

15.  Pilot's  Rating  of  Equipment  Status 

II 

9 

16.  ARINC  Research  Rating  of  Equipment 

Status 

II 

18 

TABLE  4 

f-106  shop  data  used 
IN  THIS  STUDY 


Variable  --  Code  Explained  In: 

Appendix 

Number 

Item 

Number 

1.  Flight  Line  Maintenance  Personnel 

Identification 

II 

36  & 37 

2.  Sympton  Reported  to  the  Flight  Line 

3.  Whether  or  not  the  Flight  Line  Mai- 

II  & III 

16 

function  was  Verified 

II 

19 

4.  Whether  the  Equipment  Operated 

Successfully  on  the  Next  Mission 
Following  Plight  Line  Maintenance 

II 

23 

5.  Unit  Serial  Number  Removed  at  the 

Flight  Line 

II 

27 

6.  Symptom  Reported  to  Shop  by  the 

Flight  Line  Maintenance  Crew 

IV 

-- 

7.  Diagnosis  of  Unit  Condition 

II 

33 

8.  Test  Procedure  Used 

II 

34 

9.  Repair  Action  Taken  for  Unit 

II 

33 

10 
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undoubtedly  other  differences  In  procedures  between  bases,  but 
these  do  not  seem  to  be  greater  than  differences  which  exist 
within  bases  from  time  to  time  - depending  upon  circumstances. 

Because  of  the  overall  similarity  In  symptom  reporting 
methods,  and  the  general  Inconsistency  or  unspeclflablllty,  or 
both,  of  the  differences.  It  seems  safe  to  assume  that  the  data 
analyzed  In  this  study  are  as  representative  of  current  pro- 
cedures as  Is  possible.  Generalizations  from  results  obtained 
from  these  data  analyses  are  probably  as  safe  (or  unsafe)  as 
they  would  be  from  data  collected  at  any  other  base. 

3.2  Symptom  Coding 

Symptom  coding  Is  a matter  of  some  concern  because  of  a 
problem  one  always  has  In  coding  data;  How  much  of  the  unique- 
ness of  this  observation  Is  Irrelevant  to  the  Issue  at  hand? 

The  detailed  material  In  Appendix  IV  Indicates  how  this 
problem  was  solved  for  this  study.  In  general,  each  ASB/4 
symptom  was  taken  as  a verbatim  language  specimen.  It  was 
parsed  (l.e.,  broken  down  grammatically)  Into  four  major  cate- 
gories; subject,  verb  phrase,  subject  modifiers,  and  modifying 
(or  qualifying)  words  and  phrases.  After  analyzing  each  symptom 
in  this  manner,  the  symptom  components  were  each  assigned  a four- 
digit code.  The  codes  were  assigned  In  such  a way  that  all 
grammatically  (and  semantically)  distinct  word  patterns  are  numer- 
ically distinct,  but  degrees  of  grammatical  similarity  are  repre- 
sented by  similar  degrees  of  numerical  similarity. 

This  coding  system,  which  may  be  viewed  as  an  experiment 
Itself,  worked  quite  well,  and  can  be  recommended,  with  some 
Improvements,  to  maintenance  data  processors  and  personnel  inter- 
ested In  research  using  repeated  samples  of  verbal  material  as 
dependent  variables . 
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SECTION  III 


METHODOLOGY 

The  "increasing  complexity  of  modern  military  systems" 
has  been  mentioned  so  often  It  Is  nearly  cliche,  but  a recog- 
nition of  "the  complexity  problem"  demands  not  only  Increasing 
attention  be  given  to  the  role  of  man  In  the  system,  It  also 
demands  Improved  and  Increasingly  responsive  techniques  for 
quantifying  his  performance  In  this  role.  The  present  study, 
as  part  of  an  effort  to  satisfy  a portion  of  this  need,  posits 
two  procedures  which  should  help  satisfy  this  demand.  The 
first  Is  Information  theory  as  it  Is  employed  In  this  report; 
the  second  Is  the  enumeration  of  alternative  criteria  for 
evaluating  maintenance.  These  two  procedures,  as  they  affect 
the  present  study,  are  discussed  below.  The  third  section 
below  describes  the  general  computer  analyses  prepared  for 
this  study. 

1,  Information  Theory 


Information  theory,  a relatively  new  development  In 
mathematics,  has  been  receiving  considerable  attention  In 
behavioral  research  over  the  past  decade  because  of  Its  ability 
to  handle  qualitative  data.  Its  application  In  the  context 
of  maintenance  reporting  Is  new,  and  this  section  of  the  report 
Is  Intended  to  provide  a brief  overview  of  Information  theory 
--  what  It  means  and  how  It  Is  applied. 

It  Is  not  the  purpose  of  this  section  to  derive  the 
foundations  of  Information  theory,  nor  to  expound  on  Its 
rationale.  These  topics  have  been  adequately  discussed  else- 
where (refs.  3 and  4).  The  purpose  here  Is  to  give  those 
readers  who  are  not  familiar  with  this  mathematical  technique 
a frame  of  reference  for  evaluating  the  significance  of 
analyses  using  Information  concepts. 

1.1  Fundamentals 


To  gain  "Information,"  In  the  ordinary  sense  of  that  word, 
means  that  something  must  be  learned  which  was  not  previously 
known.  In  Its  technical  sense.  Information  means  much  the  same 
thing.  In  order  to  learn  something  new  about  an  Item  or  event, 
one  must  first  have  some  degree  of  Ignorance  concerning  that 
Item  or  event.  In  the  mathematics  or  Information  theory,  this 
Ignorance  Is  labled  as  loncertalnty,  and  It  Is  defined  as  the 
amount  of  Information  which  could  be  obtained.  The  amount  of 
Information  obtained  Is  dependent  upon  the  amount  of  uncertainty 
which  previously  existed:  the  more  uncertainty  one  has  (the 
greater  the  state  of  his  Ignorance  about  something)  the  more  his 
uncertainty  can  be  reduced  (the  more  he  can  be  enlightened.) 


Now,  the  mathematics  of  Information  theory  make  It 
possible  to  measure  the  amoiint  of  uncertainty  present  (provided 
of  course  that  certain  suitable  conditions  are  first  met.) 

First,  it  Is  Important  to  see  how  lincertainty  varies  In  an 
intuitive  way,  then  the  mathematics  will  be  presented.  If  an 
equipment  operator  says  to  a maintenance  man:  "it  Is  either 
the  070  unit  or  the  6917  circuit  breaker, " then  the  maintenance 
man  has  a fair  understanding  of  what  the  problem  is  and  his 
uncertainty  about  what  has  failed  Is  rather  small.  On  the  other 
hand.  If  the  operator  had  said:  "The  performance  was  poor," 
the  maintenance  man  Is  in  a state  of  ignorance,  and  his  uncer- 
tainty about  what  has  failed  (indeed  whether  or  not  there  has 
been  a failure)  Is  relatively  high.  Similar  homey  examples  lead 
one  to  observe  that  uncertainty  Is  related  to  how  many  alter- 
natives could  have  occurred. 

Indeed,  uncertainty  Is  related  to  how  many  alternatives 
could  have  occurred.  In  particular,  the  amount  of  Information 
(H)  available  in  an  event  with  n equally  likely  alternatives 
is  defined  as  the  binary  logarithm  of  n,  or 

H = logg  n (1) 

The  choice  of  the  base  2 Is  not  arbitrary  (ref.  5).  It  produces 
a measure  which  yields  linear  relationships  when  H is  compared 
to  certain  dependent  characteristics  of  human  performance, 
such  as  errors  In  transmission  or  perception  as  a function  of 
stimulus  and  response  complexity.  Because  Information  Is 
measured  on  a logarithm  to  the  base  2 It  is  expressed  in  units 
called  "bits"  (from  binary  Information  unit). 

One  can  think  of  "Information"  as  the  smallest  number  of 
binary  (yes-no)  questions  which  must  be  asked  In  order  to 
completely  eliminate  the  uncertainty  about  an  event.  If  the 
event  has  two  equally  likely  outcomes  one  question  will  always 
work.  For  If  you  are  right,  you  know  what  the  outcome  was,  and 
If  you  then  discover  that  you  were  wrong  on  your  guess,  you 
know  what  the  outcome  was  anyway.  If  the  event  has  four  equally 
likely  outcomes.  It  would  take  you  two  questions  to  be  guaranteed 
of  finding  the  answer.  Your  question  strategy  could  be  Illustra- 
ted as  follows: 

Ask:  (First  Question) 

"Was  It  event  1 or  2?" 

If  the  answer  is  yes 

Ask:  (Second  Question) 

"Was  it  event  1?" 

If  the  answer  Is  yes,  fine. 
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If  the  answer  Is  no,  then 
it  was  obviously  event  2. 
or 

If  the  answer  (to  the  First  Question)  is  no 
Ask:  (Second  Question) 

"Was  It  event  3?" 

If  the  answer  Is  yes,  fine. 

If  the  answer  Is  no,  chen 
It  was  obviously  event  4. 

In  general,  when  all  the  outcomes  are  equally  likely,  the 
problem  of  niimber  of  questions  Is  very  easily  solved:  Logpn. 
However,  when  events  are  not  equally  likely,  uncertainty  Is 
changed  and  Information  must  be  modified.  If  a coin  Is  tossed 
ten  times  one  Is  not  surprised  to  learn  that  It  was  not  ten 
heads,  and  if  one  were  concerned  with  cnly  the  two  events: 

(1)  it  was  10  heads  or  (2)  It  was  not  10  heads,  then  one  does 
not  have  very  great  uncertainty  about  the  outcome.  Now  It  is 
difficult  to  Illustrate  how  one  might  ask  a "small  number  of 
questions",  but  It  is  easy  to  quote  the  information  theory 
formula  which  takes  this  situation  into  account: 

H = - ^ log2 

where  H is  the  total  information  in  the  set  of  outcomes. 

1 is  a counting  index  for  the  set  of  outcomes  of 
the  event . 

Pj^  is  the  relative  probability  of  each  outcome. 

This  formula  yields  the  nvimber  of  questions  one  would  have  to 
ask,  on  the  average,  if  one  used  the  very  complicated  optimum 
questioning  order  which  was  required  by  that  situation.  The 
problem  of  finding  that  order  is  another  problem  and  is  beyond 
the  scope  of  the  current  treatment. 

1.1.1  Univariate  Analysis  - An  Example 

When  N equally  likely  alternatives  can  occur,  the 
probability,  P,  of  each  alternative  is  1/N.  Hence, 

H = log2  N = logg  1/P  (3) 
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Now,  this  relationship  can  be  generalized  to  cases  in  which 
the  alternatives  are  unequally  likely: 

h^  = log2  l/P^  (4) 

where  h^  is  the  information  associated  with  the  occurrence 

of  alternative  1 and 

is  the  likelihood  of  alternative  1. 

Suppose  that  a biased  coin  comes  up  "heads"  nine  times  out  of 
ten  (P4  = .90).  According  to  this  equation  the  Information 
associated  with  a "head"  on  a particular  toss  is: 

log2  I/.90  = log2  1.11  = .15  bits 

and  for  tails  it  would  be 

log2  I/.IO  = logg  10  = 3.32  bits 

(The  disparity  between  these  nximbers  is  in  agreement  with  our 
intuitive  sense  of  uncertainty  as  described  above.  In  align- 
ment with  this  is  the  fact  that  the  informational  value  of  a 
particular  event,  h = logg  1/P,  is  called  its  surprlsal . ) 

If  these  two  informational  values  are  averaged  in  terms  of 
their  relative  likelihood  of  occurring,  the  result  is  the 
information  to  be  expected  in  the  event: 

« * I "’l*'!  = L Pi  1/Pl  (5) 

1 1 


where,  again  P^^  is  the  likelihood  of  the  event  and 

hj^  is  the  informational  value  of  the  event. 
For  computational  convenience  this  formula  can  be  reduced 

to : 

« = -I  Pi  Pi 

1 

If  the  likelihood  of  each  event  is  based  upon  the  number 
of  times  that  event  occurred,  N.,  divided  by  the  total  niimber 
of  observations,  N,  (that  is  If^P.  is  based  upon  a finite  set 
of  observations)  then 
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and  this  reduces  to; 

logg  N - ^ ^ log2  (7) 

1 

Whether  one  uses  equation  (2)  or  equation  (7)  is  a matter 
of  choice  and  computational  convenience;  the  result  is,  needless 
to  say,  the  same. 

Table  5 is  an  example  of  how  the  second  formula  (7)  is 
applied,  in  these  data,  to  measure  the  uncertainty  associated 
with  malfunction  verification.  Although  there  are  eight  possible 
alternatives,  and  thus  3*0  bits  of  possible  information,  there 
are  only  1.08  bits  of  information  actually  available.  This 
is  due  to  the  predominance  of  the  code  associated  with  reported 
verification  (there  is  very  little  surprlsal  associated  with 
the  discovery  that  a particular  malfunction  was  verified). 


These  two  calculations  can  be  related  by  yet  another 
formula  to  indicate  the  amount  of  information  already  avail- 
able, I *(that  is  information  given  by  knowing  that  the  sample 
has  unequally  occurring  alternatives); 


"s  = ^T  - ^s 


(8) 


where  I = the  information  given  by  knowledge  of  the  sample 

O 

U,p  = the  maximum  possible  information  in  this  context 

U = the  uncertainty  calculated  for  the  set  of 
outcomes  which  could  have  occurred. 

1.1.2  Bivariate  Analysis  - An  Example 


The  preceding  paragraph  set  the  stage  for  measuring  the 
amoimt  of  information  given  by  a particular  symptom  (the 
difference  between  what  was  known  before  the  symptom  was  reported, 
Um>  and  what  was  left  to  be  found  out,  U - see  section  IV, 
paragraph  1).  But  it  also  set  the  stage^for  considering 


^The  reader  need  not  be  alarmed  by  this  shift  from  H to  U and 
I,  the  quantities  are  numerically  equivalent,  only  the  point 
of  view  is  changed.  H traditionally  refers  to  information  in 
general;  U refers  to  iincertalnty  or  information  to  be  found 
out;  I is  Introduced  for  reader  convenience  as  the  corresponding 
term  for  information  already  found  out. 
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TABLE  5 


EXAMPLE  OP  INFORMATION  (UNCERTAINTY)  CALCULATION, 
MALFUNCTION  VERIFICATION 


Code 

NiJinber  of 
Occurrences 

"i 

Uncertainty  (U)  = 
Log2 

Blank 

1. 

.00055 

.00598 

1 

1428 

.78939 

.26933 

2 

0 

— 

— 

3 

31 

.01714 

.100556 

4 

199 

.11000 

.35028 

5 

0 

— 

— 

6 

31 

.01714 

.100556 

9 

119 

.06578 

.25826 

Total 

1809 

1.00000 

1 . 08496 

Verification  Codes; 


1. 

Specific  operator  complaint  verified. 

2. 

Specific  operator  complaint  not  verified, 
trouble  found  in  same  subsystem. 

Other 

3. 

Specific  operator  complaint  not  verified, 
found  in  related  or  associated  subsystems. 

Trouble 

4. 

Maintenance  originated  complaint. 

5. 

Not  checked. 

6. 

Advance  base  operation.  No  repair  data  available. 

9. 

No  trouble  found. 
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bivariate  constraint  — the  situation  where  knowledge  concerning 
one  variable  can  be  "converted"  into  partial  knov/ledge  concerning 
a second.  This  Is  very  much  like  correlation  between  quantified 
variables.  In  fact,  this  Is  so  much  like  correlation  between 
quantified  variables  that  its  application  to  qualitative  material 
has  been  something  like  a breakthrough  In  the  state-of-the-art 
of  performance  measurement . 

Information  theory  can  be  used  to  measure  the  amount  of 
knowledge  to  be  gained  about  an  event  of  concern  by  learning 
about  a collateral  event.  Thus  It  can  be  used  to  "predict"  eye 
color,  given  hair  color  or  to  predict  the  trouble  shooting 
procedure  to  be  tried,  given  the  symptom  which  was  reported. 
Clearly,  It  Is  applications  like  the  latter  which  make  Information 
theory  so  Inviting  In  the  present  context. 

In  a bivariate  analysis  one  considers  the  two  variables 
under  observation  (say  A and  B)  as  separate  characteristics 
of  one  compound  event,  (e.g.  to  compare  gender  to  eye  color 
one  simply  thinks  of  the  set  of  possible  observations:  blue- 
eyed girl,  brown-eyed  girl,  green-eyed  girl,  ...  , blue-eyed 
boy,  brown-eyed  boy,  green-eyed  boy,  ...  ) The  Information  In 
this  event  is  calculated  as  before,  using  either  equation  (2) 
or  (7),  and  the  result  is  called  the  Joint  uncertainty,  Ua&B 
If  this  measure  is  compared  to  Ua,  the  uncertainty  in  variable 
A,  plus  Ub,  the  uncertainty  In  variable  B,  the  result  is  an 
assessment  of  the 


where  Ug.p  = 


^S&P  " 

Figure  2 represents  the  application  of  equation  (9)  to 
F-106  data  corresponding  to  pilot  and  symptom.  The  degree  of 
constraint  is  2.42205  bits  of  Information.  Unfortunately,  this 
example  Is  not  simple.  When  comparing  the  extent  to  which  a 
pair  of  variables  overlap,  the  amount  of  Joint  information  (Ua&b) 
must  be  compared  to  the  total  amount  of  Information  possible. 

This  latter  Is  limited  by  both  the  sample  size  (the  number  of 


degree  of  overlap  between  these  two  variables. 


^S:P  “ ■ ^S&P 


(9) 


the  am.ount  of  uncertainty  about  S which 
remains  given  knowledge  of  P.  (the  constraint) 

the  a priori  amount  of  uncertainty  associated 
with  variable  S. 

the  a priori  amount  of  uncertainty  associated 
with  variable  P. 


the  amount  of  uncertainty  associated  with  the 
compound  event  (S&P) 
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^S&P 


(Area  = 5.29164 

(Area  = 6.10230 

^S&P  (AreaO)  = 8.97189 

^S:P  (Area  # ) = Ug  + Up  - Ug^p  (Eq.  9) 

= 2.42205 

FIGURE  2 

EXAMPLE  OF  JOINT  UNCERTAINTY  CALCULATION 
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different  observations  possible!  and  the  sum  of  the  separate 
a priori  uncertainties  (Ua  + Ub) . In  the  present  sample,  there. 


to  variable  A:B.This  term,Cy^;B,  is  very  much  like  r2  for  product 
moment  correlation  and  thaf^accounts  for  the  possibility  of 
using  the  square  root  of  Ca;B  an  estimate  of  the  degree  of 
correlation  between  variables  A and  B (ref.  6). 

Ca;b  is  read  "The  constraint  of  variable  A by  variable  B." 
In  the  example  given  in  Figure  2 pilot  constrains  symptom, 

Cs:p,  by  35.8^  and  symptom  constrains  pilot,  Cp.3,  by  29.5^. 

The  former  constraint,  the  more  Interesting,  suggests  that, 
for  F-I06  avionics  malfunctions  reported  at  the  flight  line, 

36^  of  the  variability  in  what  is  wrong  can  be  accounted  for 
(predicted)  by  who  flew  the  airplane  I 

1.2  Applications 

In  the  present  study.  Information  theory  is  applied  in 
two  ways.  First,  as  a univariate  calculation:  here,  for 
example,  one  counts  the  niimber  of  times  each  particular  type 
of  "black  box"  is  handled  in  connection  with  a certain  symptom*. 
The  Information  contained  in  the  distribution  of  occurrences 
for  these  black  boxes  is  then  calculated.  If  this  Information 
decreases  when  words  are  added  to  the  symptom  in  question,  tKai 
the  additional  words  reduced  the  \incertalnty  concerning  which 
■'black  box"  is  the  bad  one.  That  is  to  say,  the  additional 
words  convey  information.  In  particular  they  convey  exactly 
the  amount  of  information  by  which  the  uncertainty  changed. 


*Symptom  here  must  mean  a set  of  observations:  otherwise  there 
is  no  distribution  of  units  to  count.  The  set  is  defined  in 
terms  of  reported  words  or  word  patterns  which  are  the  same, 
or  which  appear  to  be  the  same.  As  it  happens,  the  change  of 
information  as  the  definition  of  sameness  changes,  is  taken 
as  a measure  of  sameness. 


are  only  I809  observations;  thus  Ut  (or  Umax)  is  less  than  or 
equal  to  Log2(l809)  =10.8.  Adjusting  Us:p  to  compensate  for 
this  lower  maximum  possible  uncertainty"  yields  what  seems  to 
be  a conservative  estimate  of  overlap,  I.80  bits. 

No  matter  how  U^-b  Is  calculated,  whether  by  comparison 
with  log2  N or  with  Ua  + Up,  the  fact  remains  that  it,  in  turn 
can  be  compared  with  both  and  Up.  If  one  draws  the  quotient 


compared 
^A:B  ^B" 


U 


''A:B 


A:B 


(10) 


he  obtains  a measure  of  the  percentage  of  variability  of 
characteristic  B which  can  be  accounted  for  by  relationship 
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The  second  application  of  Information  theory  Is  In 
bivariate  analysis.  Here,  Information  theory  Is  used  as  a 
technique  for  correlating  qualitative  data.  Each  of  the 
variables  In  question  Is  considered  as  being  a set  of  observa- 
tions sampled  from  a large  population  with  a finite  niomber 
of  different  states.  (An  observed  data  point  could  be  any  one 
of  a finite  nvimber  of  'Veadlngs".)  The  \ancertalnty  In  the 
distribution  of  these  states  Is  calculated  for  each  variable 
separately.  When  the  sm  of  these  uncertainties  Is  compared 
to  the  uncertainty  calculated  for  the  compoiind  event,  (the 
variables  In  question  being  taken  as  a pair)  a measure  of  the 
degree  to  which  either  variable  constrains  the  other  Is 
obtained. 

Information  theory  Is  a relatively  recent  mathematical 
development  and  the  meaning  of  Its  "sampling  distribution" 

Is  not  yet  defined.  As  a result,  there  Is  no  direct  statis- 
tical method  for  evaluating  statistical  significance.  In  the 
present  study,  since  concern  Is  with  "practical"  rather  than 
"statistical"  significance,  this  Is  not  particularly  disarming. 
However,  It  would  be  nice  to  have  a procedure  for  evaluating 
the  chance  likelihood  associated  with  any  particular  observation. 

In  order  to  estimate  the  sampling  distribution  for  univariate 
analysis  of  the  ASB/4  data,  a number  of  random  samples  of  data 
were  selected  and  the  univariate  uncertainties  were  calculated. 
Figure  3 plots  the  observed  mean  of  these  calculations  as  a 
function  of  sample  size.  Figure  3 also  Includes  the  maximum 
possible  Information  (logp  N;  and  the  observed  10th  percentile 
values.  Once  a specif led^sample  of  data  is  described,*  its 
\incertalnty  can  be  compared  to  that  Indicated  by  the  Figure 
for  samples  of  equivalent  size.  (The  calculated  values  on 
this  Figure  are  probably  spuriously  low  because  random  samples 
from  the  ASB/4  data  Include  a large  nvimber  of  preventive  main- 
tenance actions.  These  actions  are  not  associated  with  "black 
box"  activity,  and  thus  the  particular  black  box  "no  vinlt 
removed"  Is  a likely  observation.  This  being  so.  Figure  3 
Is  probably  a conservative  test  when  used  to  evaluate  informa- 
tion transmission  in  operator  malfunction  reports.) 

In  order  to  determine  the  statistical  unlikelihood  of  a 
bivariate  calculation,  an  "efficiency  measure"  of  bivariate 
constraint  was  defined.  This  measure  attempts  to  assess  the 
usefulness  of  a given  amount  of  bivariate  constraint.  For 
example.  If  there  are  1024  observations  of  two  variables,  there 
Is  a possible  uncertainty  of  10  bits.  If  one  variable  has 


*A  sample  of  data  now  becomes  a set  of  observations  hypothesized 
to  be  repeated  reports  of  a single  symptom. 


6 bits  by  itself,  a second  has  3 bits  by  Itself,  and  If  the 
Joint  event  has  8 bits  of  Information,  then  the  first  event 
constrains  the  second  event  by  one  bit  (6+3  - 8)  or  33^  (one 
bit  of  constraint  divided  by  the  three  bits  in  the  constrained 
variable).  This  is  an  efficiency  of  16.65^  (one  bit  of  constraint 
divided  by  the  six  bits  in  the  constralng  variable.)  If  all 
of  the  data  (10  bits)  were  used  to  predict  the  second  variable 
(all  of  the  variable's  uncertainty  is  always  predictable  from 
all  of  the  observations),  the  efficiency  would  be  30^  (three 
bits  of  constraint  divided  by  ten  bits  of  observation.)  Where 
we  predicted  one  bit  using  six  bits,  the  observed  constraint 
(of  33^)  is  not  really  very  Impressive,  in  spite  of  its  absolute 
size,  because  it  predicts  with  less  efficiency  than  could  be 
achieved  by  using  all  the  data.  Somehow  it  seems  that  a good 
prediction  could  do  "better"  than  this  sort  of  an  average. 

(This  is  not  a rigorous  concept  and  has  been  offered  only  as 
a tentative  method  of  evaluating  the  efficiency  of  observed 
bivariate  constraint.) 

2.  MEASURES  OF  MAINTENANCE  EFFICIENCY 

Maintenance  efficiency  can  be  measured  in  a variety  of 
ways.  And  if  concern  is  really  with  the  efficiency  of  main- 
tenance, then  all  reasonable  measures  should  be  explored  and 
compared  The  present  study  uses  five  criteria.  The  first, 
the  mos.  familiar  criterion  - maintenance  time,  is  subdivided 
into  seven  categories:  (See  Table  2 item  8 and  Table  3 item 
14) 

1.  Preparation  (Prep.)  Time  The  clock  hours  necessary 

for  the  maintenance  man  to 
get  ready  to  work  on  the 
system,  to  check  paper 
records,  to  obtain  tools, 
to  warm  up  equipment,  etc. 

2.  Malfunction  Verification  The  clock  hours  necessary 

(MV)  Time  for  the  maintenance  man  to 

verify  that  the  system  has 
Indeed  failed  in  a manner 
consistent  with  the  reported 
symptom. 

3.  Fault  Location  (F.L.)  The  clock  hours  necessary 

Time  for  the  maintenance  man  to 

locate  the  malfunctioning 
black  box. 

4.  Part  Procurement  (P.P.)  The  clock  hours  necessary 

Time  for  the  maintenance  man  to 

obtain  the  necessary 
replacement  units. 
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5.  Repair  (Rep.)  Time 


6.  Malfunction  Administra- 
tive Delays  (Malf. 

Ad . ) Time 


7.  Total  Subsystem  Down 
Time 


The  clock  hours  necessary 
for  the  maintenance  man  to 
physically  repair  the  system, 
remove  and  replace  units, 
adjust  units,  check  out  the 
final  repair,  etc. 

The  clock  hours  necessary 
for  the  maintenance  man  to 
perform  other  duties,  e.g., 
lunch,  coffee,  another  Job, 
sick  call,  etc. 

The  total  clock  hours  the 
system  was  out  of  service. 


A second  familiar  criterion  Is  the  probability  of 
malfunction  verification.  An  unverified  malfunction  Is  bad,  for 
If  the  system  has  really  malfunctioned,  failed  equipment  Is  put 
Into  service  as  though  It  were  good.  On  the  other  hand.  If 
the  system  has  not  really  malfunctioned,  then  maintenance 
personnel  have  wasted  their  time  (and  perhaps  Introduced  fail- 
ures*) and  the  system  has  needlessly  been  out  of  use.  (See 
Table  2,  Item  5j  and  Table  3>  Item  6). 

The  third  criterion  Is  analogous  to  the  second.  It 
Involves  verification  in  the  shop  rather  than  on  the  flight 
line  and  Is  related  to  wasted  effort  at  the  shop  level.  "Bench 
Check  OK"  is  the  probability  that  a black  box  which  a flight 
line  technician  has  pulled  from  an  aircraft  during  repair  will 
check  as  operating  satisfactorily  when  tested  by  shop  mainte- 
nance personnel.  (See  Table  4,  Item  7.) 

The  fourth  criterion  used  here  Is  next  flight  success. 

Was  the  equipment  (here  the  P-106  avionics)  successfully 
utilized  on  its  next  attempted  mission?  This  Is  not  only  a 
measure  of  maintenance,  it  surely  Includes  a component  of 
operator  efficiency.  (See  Table  3>  Item  13.) 

The  final  criterion  is  the  amount  of  Information  transmitted 
by  reported  symptoms.  If  one  man  says  more  in  his  symptoms  than 
another  man,  he  is  clearly  reporting  more  efficiently.  (See 
Section  III,  paragraph  1.2). 


*A  recent  study  of  B-58  maintenance  data  suggests  that  reliability 
of  good  equipment  can  be  reduced  up  to  20^  when  It  Is  maintained 
unnecessarily  (ref.  7). 


3 . DATA  PROCESSING 

As  is  becoming  common,  most  of  the  calculations  required 
for  this  study  were  accomplished  by  computer.  Three  programs 
(See  Appendix  V)  were  prepared  for  the  on-site  IBM  1401. 

The  first  program  calculated  the  uncertainty  in  black 
boxes  mentioned  in  "little  decks"  of  ASB/4  data.  The  original 
data,  as  coded  and  punched,  was  sorted  (according  to  any 
sensible  criterion)  into  piles  of  cards,  or  "symptoms."  These 
little  decks  were  separated  by  blank  cards  and  run  through  the 
machine  one  after  another.  The  program  yielded  an  uncertainty 
measure  for  each  little  deck  and  an  unweighted  average  for  the 
whole  set. 

The  second  prograiTi  used  the  same  input  as  the  first 
program,  but  treated  the  problem  as  a simple  randomized  design 
analysis  of  variance  for  each  of  the  six  categories  of  time 
and  also  for  the  nvimber  of  men  in  each  of  the  three  (3,  5,  & 7) 
skill  levels.  (Skill  level  and  numbers  of  personnel  at  a skill 
level  was  Included  in  the  analysis  as  an  exploratory  variable. 

It  is  not  certain  whether  these  variables  function  as  criteria 
of,  or  are  dependent  upon,  Job  complexity.) 

The  third  program  was  similar  in  intent  to  the  first,  but 
more  general.  It  was  used  in  two  separate  machine  passes  for 
calculating  bivariate  information.  Here  the  data  cards  are 
sorted  (alpha-numberic  fields)  on  any  pair  of  dimensions. 

(The  dimensions  can  be  any  two  sets  of  data  colxomns  located 
anywhere  on  the  data  card  --  the  only  limitation  is  that  subse- 
quent machine  comparison  of  observations  was  limited  to  two 
sets  of  five  consecutive  fields.  Careful  sorting  can  allow 
these  fields  to  be  adequate  for  multivariate  analysis  of  three, 
four,  or  even  five  variables.)  This  sorting  operation  amounts 
to  converting  the  data  into  an  ordered  comparison  matrix,  one 
observation  at  a time.  The  Phase  I program  accepted  these 
cards,  and  a header  card  containing  a description  of  the  card 
fields  to  be  compared,  and  produced  as  punched  output  an  ordered 
comparison  matrix,  one  cell  at  a time.  The  Phase  I program  also 
coiinted  the  nimiber  of  non-empty  data  cells  and  punched  this 
ntimber.  Phase  II  accepted  the  output  of  Phase  I as  input. 

This  phase  calculated  the  uncertainty  of  one  of  the  variables 
under  investigation  (the  uncertainty  in  the  column  sums  of  the 
data  matrix)  and  also  the  uncertainty  of  the  two  variables 
combined  (the  uncertainty  of  the  cell  sums  of  the  data  matrix). 
To  obtain  the  uncertainty  associated  with  the  second  variable 
being  studies,  a second  application  of  the  Phase  II  program 
was  required.  Properly  planned,  this  second  application  could 
be  in  conjunction  with  another  variable,  and  an  additional 
combined  \incertalnty  could  be  obtained.  In  this  manner  it  is 
possible  to  study  the  total  mutual  structure  of  three  variables 
in  three  passes  of  the  Phase  II  program. 


SECTION  IV 


ANALYSIS 

The  subsections  of  this  section  present  the  results  of  the 
various  analyses  conducted  during  this  study.  The  subsections 
parallel  the  analyses  themselves, which  attended  to  three  separate 
Issues : 

(1)  VJhere  is  the  information  contained  in  symptoms? 

(2)  What  can  be  predicted  from  symptom  content? 

(3)  Can  information  theory  be  applied  to  maintenance 
data  in  general? 

1.  INFORMATION  IN  SYMPTOMS 

The  fundamental  assumption  here,  and  Indeed  the  fundamental 
assumption  behind  malfunction  reporting  in  any  context,  is  that 
symptoms  contain  information.  They  say  what  is  wrong,  which 
"thing  has  busted."  They  direct  behavior.  While  it  is  necessary 
to  check  that  assumption,  it  is  also  a purpose  of  this  study  to 
know  what  part(s)  of  the  ssrmptom  are  useful. 

To  prove  this  assumption  is  trivial:  maintenance  people 
don't  do  maintenance  without  a symptom,  therefore  the  occurrence 
of  a symptom  is  an  excellent  predictor  of  the  occurrence  of  a 
maintenance  action.  To  measure  the  amount  of  information  in  a 
symptom  is  not  trivial.  This  is  the  primary  subject  of  the 
following  paragraphs. 

1.1  A Bench  Mark 


Before  the  amount  of  Information  given  by  a particular 
symptom  can  be  calculated,  how  much  information  could  have  been 
given  must  be  known.  Then  the  amount  which  given  is  merely 
the  difference  between  what  could  have  been  given  and  how  much 
had  to  be  found  out  by  troubleshooting.  In  notation,  this  is 
simply: 


Ut  = Is  + U3  or  I3 


U. 


u 


s 


where 

= the  total  uncertainty  in  the  system  (see  below) 

I = the  information  contained  in  the  symptom  (the 
information  given  to  the  maintenance  man)  and 

U = the  uncertainty  remaining  in  the  symptom  (the 
uncertainty  contained  in  the  distribution  of 
the  units  handled  in  the  set  of  maintenance 
actions  used  to  define  the  symptom). 
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The  ASB/4  has  l46  flight  line  replaceable  units.  Thus, 
there  are  147  black  boxes  which  a technican  can  remove  on  a 
given  maintenance  action  (including  the  black  box  "remove 
nothing").  Accordingly,  there  is  a total  of  7.20  bits  (Log2  147) 
of  uncertainty  possible  concerning  which  black  box  was  removed 
on  any  particular  maintenance  action.  When  the  relative  failure 
rates*  of  the  boxes  in  the  ASB/4  (sp  log  p)  are  considered,  the 
uncertainty  as  to  which  particular  black  box  was  removed  on  a 
particular  maintenance  event  is  only  5.93  bits. 

This  latter  number  can  serve  as  Ut  - the  bench  mark  for  sub- 
sequent calculations.  If  a symptom,  as  defined  in  a particular 
case,  yields  a distribution  of  units  maintained  with  an  un- 
certainty of  less  than  5*93  bits,  then  the  difference  in  un- 
certainties is  the  measure  of  the  information  in  that  symptom. 

1.2  An  Assumption 

The  number  calculated  above  is  probably  a reasonable  estimate 
of  uncertainty  in  this  context,  but  the  reader  must  be  warned  that 
the  following  calculations  assume  that  removal  rates  are  equiva- 
lent to  failure  rates . (If  this  is  a bad  assumption  the  results 
of  this  study  will  err  on  the  conservative  side'.)**  In  the  utility 


* The  "relative  failure  rate"  of  the  box  "remove  nothing"  is  the 
relative  likelihood  of  removing  nothing.  (It  may  be  of  passing 
interest  to  note  that  22  of  the  l46  removeable  units  did  not 
fail  during  the  data  collection  period. 


**  The  reason  for  this  is  not  simple.  In  the  first  place,  analysis 
of  removal  rates  and  shop  check  OK  rates  indicates  that  for  at 
least  the  F-106  flight  line  maintenance  men,  removal  rates  tend 
to  be  correlated  to  failure  rates  and  to  exceed  failure  rates 
by  an  amount  proportional  to  the  failure  rate.  That  is,  these 
maintenance  men  seem  to  be  removing  units  too  often  but  in 
proportion  to  the  failure  rates.  Thus  an  uncertainty  calculated 
for  removal  rates  would  be  nearly  the  same  as  an  uncertainty 
calculated  for  failure  rates.  In  the  second  place,  if  removal 
rates  were  not  correlated  to  failure  rates,  then  maintenance 
men  would  probably  be  troubleshooting  to  optimize  efficiency, 
that  is  they  would  always  remove  the  unit  most  likely  to  cause 
the  symptom;  and  only  if  that  failed  to  clear  up  the  symptom 
would  they  remove  another  unit.  This  would  produce  an  un- 
certainty calculated  on  removal  rates  which  was  less  than  the 
uncertainty  based  on  failure  rates.  (The  latter  being  what 
the  symptom  really  communicates.)  Now,  if  Us  is  deflated,  then 
Is  is  inflated  and  subsequent  efforts  to  Increase  Is  by  adding 
elements  to  the  symptom  are  handicapped.  Anything  which  in- 
creased this  Is  would  surely  have  Increased  the  lower  Is 
derived  from  the  higher  Ug  based  on  failure  rates. 
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structure  of  peace  time  military  maintenance,  this  is  probably  a 
fairly  sound  assumption.  It  is  predicted  by  psychological  prob- 
ability choice  theory  and  it  is  consonant  with  the  personal 
opinions  of  many  civilian  maintenance  analysts.  It  is  not  the  way 
things  should  be,  but  it  appears  to  be  the  way  things  are.  (Since 
the  P-106  data  Included  shop  follow  up  data,  it  is  possible,  for 
selected  symptoms,  to  compare  bench  check  okay  rates  versus  bench 
check  bad  rates  for  groups  of  maintenance  personnel  over  a period 
of  several  similar  maintenance  actions.  For  certain  simple  flight- 
line  troubleshooting  strategies  it  is  computationally  easy  to 
predict  what  these  relative  rates  ought  to  be  in  view  of  the  real 
failure  rates- (as  determined  by  the  shop's  bench  check  results). 
When  this  is  done  for  real  P-IO6  data  (ref.  8),  it  is  clear,  at 
statistical  significance  levels  well  beyond  .001,  that  flight  line 
personnel  are  minimizing  neither  system  down  time  nor  shop  main- 
tenance time.  Unfortunately,  it  was  not  computationally  feasible 
to  predict  what  these  relative  rates  would  be  for  maintenance 
personnel  who  matched  removal  rates  to  failure  rates.  However, 
the  fact  that  maintenance  men  are  not  using  the  rational  optimum 
trouble-shooting  strategies,  does  tend  to  Increase  the  credibility 
of  the  stulated  suboptimum  procedure.) 

1 . 3 The  Number  of  Phrases 

Perhaps  the  easiest  thing  to  measure  about  a verbal  descrip- 
tion is  its  length,  and  it's  an  obvious  question  to  wonder  what 
effect  the  number  of  phrases  in  a ssnnptom  might  have  upon  main- 
tenance behaviors.  When  an  operator  says  more,  does  he  really 
communicate  more? 

Table  6 shows  the  amount  of  uncertainty  (Ug)  associated 
with  the  number  of  phrases  per  symptom  for  ASB/4  flight  line 
symptoms.  Two  and  three  phrase  symptoms  are  more  uncertain 
concerning  which  unit  is  removed  than  one  phrase  symptoms.  There 
are  two  primary  reasons  for  this:  The  first  is  that  the  one- 
phrase  symptoms  contain  a much  larger  percentage  of  periodic 
maintenance  symptoms  than  the  multiphrase  symptoms.  Since  periodic 
maintenance  symptoms  have  a very  low  uncertainty  (l.e.  are 
associated  with  a very  small  number  of  units)  this  tends  to  change 
the  unit  frequency  distribution  of  the  one-phrase  symptoms,  and 
thus  it  lowers  the  uncertainty.  A second  reason  that  the 
probability  of  malfunction  verification  increases  with  the  number 
of  phrases  in  a symptom  (See  Table  7).  Since  unverified  symptoms 
usually  have  no  removed  units  associated  with  them,  and  since 
"no  unit"  is  treated  as  a single  unit  category  in  the  information 
calculation,  this  also  tends  to  depress  the  uncertainty  in  one- 
phrase  symptoms.  It  is  possible  to  check  these  assumptions  by 
performing  the  information  calculation  without  periodic  maintenance 
symptoms  or  unverified  symptoms . The  assumptions  prove  to  be 
tenable,  as  can  be  seen  in  Table  8. 
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TABLE  6 

UNIT  REMOVED  VERSUS 
UNCERTAINTY  IN  NUMBER  OP  PHRASES 

asb/4  data 


No.  of  Phrases 

N 

Uncertainty 

1 

1434 

4.1118 

2 

222 

4.7692 

3 

47 

4.3426 

4 or  more 

19 

3.9250 

TABLE  7 

PROBABILITY  OF  MALFUNCTION  VERIFICATION 
AS  A FUNCTION  OP  NUMBER  OP  PHRASES 

asb/4  data 


No.  of  Phrases 

Probability  of  Verification 

1 

.475 

2 

.592 

3 

.656 

4 or  more 

.937 

TABLE  8 


UNCERTAINTY  IN  UNIT  REMOVAL  VERSUS  NUMBER  OF  PHRASES 
VERIFIED  OPERATOR  COMPLAINTS  ONLY 

asb/4  data 


No.  of  Phrases 

N 

Us 

Is 

1 

711 

5.84 

.09 

2 

120 

5.18 

.75 

3 

28 

4.01 

1.92 

4 or  more 

22 

3.64 

2.29 
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As  might  be  expected,  uncertainty  is  reduced  (information  is 
increased)  when  symptoms  become  longer  and  longer.  This  decrease 
in  uncertainty  suggests  that  increased  length  is  functionally 
equivalent  to  increased  specificity.  In  particular,  it  indicates 
that  knowledge  of  how  many  phrases  were  used  is  equivalent  to  a 
small  amount  of  Information  concerning  what  will  be  removed. 

Some  care  must  be  taken  in  interpreting  the  results  of 
analyses  involving  number  of  phrases,  since  an  additional  phrase 
is  often  a qualifying  phrase  indicating  either  an  Intermittency 
or  some  specificity  in  the  occurrence  of  a symptom.  (See  below 
and  Section  IV,  paragraph  I.9  for  an  explanation  of  the  impli- 
cations of  this  interpretation.) 

An  examination  of  F-I06  shop  symptoms  shows  that  "bench  check 
OK"  rates  go  up  (probability  of  verification  goes  down)  as  a 
function  of  the  number  of  phrases  or  the  number  of  words.  (This 
is,  of  course,  counter  to  flight  line  trend.)  This  suggests  that 
an  increase  in  flight  line  verification  as  a function  of  ssrmptom 
length  may  be  a function  of  a percleved  rather  than  an  actual 
increase  in  information.  This  may  mean  that  the  addition  of 
phrases  (or  words)  adds  a certain  amount  of  credibility  to  what 
otherwise  might  be  a suspected  "false  alarm.  " This,  in  turn,  may 
cause  maintenance  men  to  take  corrective  action  in  situations 
where  they  would  not  ordinarily  do  so,  thus  leading  to  higher 
bench  check  OK  rates.  Currently  one  cannot  determine  whether 
longer  symptoms  actually  have  more  information  as  indicated  by 
the  uncertainty  measure  or  whether  this  measure  is  an  artifact 
of  the  "credibility  effect"  of  a long  symptom. 

1 . 4 The  Subject 


The  subject  of  the  symptom  is  the  central  noun  phrase  of  the 
verbal  description  given  by  the  operator.  It  is  not  necessarily 
the  grammatical  subject  (even  though  almost  all  symptoms  are  cast 
in  active  sentence  frames.) 

The  amount  of  information  given  by  the  subject  word  of  a 
symptom  can  be  estimated  by  using  the  average  Us  for  all  subject 
codes  fsee  Appendik  IV).  The  average  uncertainty  in  2-dlgit 
subjects  is  2.0801  bits.*  When  this  is  compared  to  the  bench 


* Two  digit  subjects  are  coded  references  to  subsystems  of  the 
ASB/4.  Three  digit  subjects  were  specific  functional  items  and 
four  digit  subjects  were  unique  word  forms.  The  variety  of 
three  and  four  digit  subjects  made  the  calculation  of  average 
uncertainty  over  more  than  two  digits  an  ill-advised  estimation 
procedure.  (The  typical  three  digit  subject  occurred  fewer 
than  six  times.) 
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mark  uncertainty,  5.93  bits,  knowledge  of  subjects,  even  at  the 
gross  2-dlglt  level  of  classification,  results  In  a substantial 
gain  In  information  (3.85  bits,  to  be  exact).  The  average  sample 
size  for  2-dlgit  subjects  is  53.58  cases.  Comparing  the  obtained 
average  uncertainty  with  the  uncertainty  obtained  from  random 
data  samples  of  the  same  size  (See  Figure  2),  It  Is  obvious  that 
this  obtained  average  Is  lower  than  one  might  expect  by  chance. 

The  obtained  2.0801  bits  Is  lower  than  the  random  10^  value  of 
about  2.6  bits,  and  considerably  lower  than  the  mean  random  value 
of  about  3.3  bits.  There  can  be  no  doubt  that  subjects  convey 
Information . 

1 . 5 The  Description 

The  description  Is  the  verb  phrase  of  the  symptom,  when  there 
is  a verb  phrase;  on  other  occasions  it  is  the  nomlnallzed  verb, 
the  predicate  adjective,  or  simple  adjectival  form  employed  by  the 
operator  in  his  report.  In  the  following  symptoms  the  description 
is  underlined. 


Could  not  go  Into  short  range 
Double  VRM 


Run  ECM  Tie  In 

Poor  video  operation  30  miles 
Tg  would  not  drive  off  zero 
Stab  Weak 


Following  an  analysis  like  that  in  the  previous  paragraph, 
for  similar  reasons,  the  average  information  in  2-dlgit  descrip- 
tions is  2.7981  bits  based  on  an  average  sample  size  of  35.62 
cases  per  description.  This  is  higher  than  the  10^  random  value 
of  2.3  bits  and  quite  close  to  the  random  mean  of  3*0  bits  for 
samples  of  this  size.  Thus,  there  is  much  less  information  in 

description  no  more  than  might  be  expected  by  chance  if  the 

data  were  grouped  into  samples  of  this  size. 


Qualitatively,  there  are  two  kinds  of  descriptions  which 
appear  in  these  data.  The  first  kind  is  a technical  verb  or 
phrase  (e.g.  "drops  out  of  radiate,"  "spoking"),  which  occurs 
relatively  infrequently  in  the  language  as  a whole  but  quite 
frequently  here.  The  second  kind  of  description  is  a nomlnallzed 
adjective  or  a simple  verb  quite  common  in  the  language  as  a 
whole.  Examples  of  this  kind  of  description  are  verbs,  such  as 
"bad",  "broken",  "burned",  "bent",  "off",  and  "out".  It  turns 
out  that  the  first  kind  of  description  occurs  with  certain  subjects 
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only,  and  thus  is  redundant  l.e.  highly  correlated  with,  or 

constrained  by,  the  subject.  The  second  kind  depends  upon  the 
subject  to  give  it  meaning  (in  this  context)  and  cannot  convey 
very  much  information  in  isolation,* 

1.6  The  Modifier 


The  modifiers  are  all  those  little  fragments  of  detail, 
qualitative  and  restrictive,  which  the  operator  appends  to  his 
malfunction  observations.  This  Includes  such  things  as  when  and 
where  the  symptom  was  observed,  what  alternatives  were  attempted, 
and  how  big  an  error  was. 

Modifiers  do  not  carry  a great  deal  of  Information  in  them- 
selves (no  more  than  chance)  but  their  presence  in  conjunction 
with  a subject  or  description  does  effect  verification  rates. 

(See  Section  IV,  paragraph  2.3  below.) 

1 . 7 The  Mood 


The  mood  of  a symptom  is  more  or  less  defined  as  the  general 
tone  of  the  complaint.  It  seems  reasonable  to  ask  if  mood  is  a 
variable  affecting  information  transmission.  Accordingly,  all 
ASB/4  descriptive  statements  were  sorted  into  the  following  three 
categories : 

I.  Negative  Auxiliary  and  Description 

Negative  comment  with  or  without  additional 

remarks  (e.g.:  Bad,  could  not  , did  not  , 

has  no  , . , . ) 

II.  Neutral  Auxiliary  and  Description 

Neutral  comment  with  or  without  additional 
remarks  (e.g.:  Breaks  up,  came  on,  deflect  in, 
drops  out,  . . . ) 

III.  No  Auxiliary  or  Negative  Description 

One  word  objective  statement  of  condition 
(e.g.:  Arcs,  bent,  check,  due,  flat  hits,  ...) 

In  order  to  most  adequately  allow  an  opportunity  for  any 
effect  to  be  detected,  only  operator  complaints  were  Included 
in  this  analysis.  The  wording  of  maintenance  initiated 


* A bivariate  analysis  of  subject  versus  description  upheld  these 
general  conclusions. 


complaints  would  have  a different  effect  upon  the  behavior  of 
the  maintenance  man.  (It  doesn't  make  much  "sense"  to  talk 
about  information  transmission  within  a maintenance  man.) 

The  uncertainty  associated  with  these  three  groups  of 
symptoms  is  4.4957;  4.4374;  and  4.3969  respectively,  with  a 
total  of  4.80083  for  all  of  the  data  used  in  this  test.  This 
yields  an  Information  transmission  of  .305I;  .3634;  and  .4039. 
These  transmissions  are  only  slightly  better  than  zero,  and  are 
certainly  of  no  practical  significance,  even  though  differences 
in  transmission  vary  from  10  to  30^.  It  seems  quite  defensible 
to  claim  that  mood  has  very  little  to  do  with  what  is  being 
reported.  Apparently  there  are  no  predominant  symptoms  which 
cause  emotional  involvement.  Similar  negative  results  were 
obtained  from  an  analysis  of  the  mood  of  the  F-I06  shop  symptoms. 

1 . 8 The  Severity 

Severity  means  that  the  operator  of  the  equipment  added 
something  to  his  verbal  report  that  maded  it  a stronger  statement; 
e.g.,  "very",  "excessive",  "severe",  "too  much".  It  seemed 
reasonable  to  ask  what  Impact  this  characteristic  would  have 
upon  information  transmission.  The  question  cannot  be  answered 
simply  by  separating  all  mulf unction  reports  into  two  groups: 
neutral  and  severe.  Ther'^  are  too  many  would-be  neutral  S3nnptoms 
which  could  not  be  stated  in  severe  terms.  Accordingly  a matched 
pairs  technique  was  attempted.  Unfortunately,  there  are  only 
seven  matched  data  points.  These  are  not  enough  to  analyze. 

1 . 9 The  Intermlttency 

By  carefully  sorting  through  the  data,  it  is  possible  to 
define  a number  of  symptoms  containing  the  word  "Intermittent" 
which  are  matched  by  symptoms  which  are  Identical,  except  for 
the  presence  of  the  word  Intermittent.  For  example: 

Intermittent  Symptom  Stable  Counterpart 

Inoperative  intermittently  Inoperative 

Intermittently  Wavy  Wavy 

Intermittent  Spoking  Spoking 

An  Infoimation  analysis  of  these  pairs  with  all  of  the 

intermittent  symptoms  and  all  of  the  stable  symptoms  grouped 

together  to  form  two  symptoms  showed  that  the  intermittent 

symptoms  are  more  uncertain  (contain  less  information)  than  the 
stable  symptoms.  There  were  4.29  bits  of  uncertainty  in  the 
intermittent  symptoms  and  4.11  bits  in  the  stable  symptoms.  This 
difference  occured  in  spite  of  the  fact  that  the  smaller  sample 
size  (63  vs.  210)  of  the  set  of  intermittent  symptoms  would  tend 
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to  wash  this  effect  out.  The  effect  may  have  been  due  to  the 
difference  in  verification  races  for  these  two  groups.  Symptoms 
with  the  word  "intermittent"  in  them  have  a verification  rate 
which  is  considerably  higher  than  that  for  the  matched  stable 
symptoms.  (See  paragraph  4.2.3). 

1 . 10  The  Grammatical  Aspects 

It  seems,  from  the  previous  inf ormation^ that  of  the 
grammatical  elements  of  symptoms  (Section  IV,  paragraphs  1.3» 

1.4,  1.5,  and  1.6),  subjects  carry  most  of  the  information  in 
symptoms.  Other  functional  elements  (verbs  and  modifiers) 
serve  to  provide  specificity  or  redundancy  when  used  with 
subjects,  but  carry  relatively  little  information  by  themselves. 
In  the  language  at  large,  psychological  studies  have  shown  that 
the  noun  positions  in  sentences  carry  more  information  than  verb 
or  adjective  positions.  Some  work  has  also  shown  that  nouns 
constrain  verbs  and  adjectives  more  than  they  are  constrained  by 
them.  The  same  principles  which  govern  the  distribution  of 
Information  in  the  English  language  operate  here,  as  might  be 
expected.  There  can  be  no  doubt  that  symptoms  do  contain 
information  and  little  doubt  that  most  of  the  information  about 
which  black  box  is  bad  is  contained  in  the  subject . (It  is  sad 
that  limited  sample  sizes  prevented  an  assessment  of  the  effect 
of  the  various  ways  of  stating  the  same  subject.  Currently  one 
is  ill  able  to  say  what  the  subject  really  is.) 

1.11  The  Semantic  Aspects 

Prom  the  three  paragraphs  dealing  with  semantic  elements 
(Section  IV,  paragraphs  1.7,  1.8,  and  1.9),  the  connotative 
aspects  of  symptoms  might  appear  relatively  unimportant.  While 
this  is  strictly  true  about  information  transmission  to  the 
maintenance  man,  it  is  NOT  true  about  the  information  which 
maintenance  personnel  seem  to  imply  is  present.  This  will  become 
obvious  in  paragraph  2. 

2.  PREDICTION  BY  SYMPTOMS 

Symptoms  convey  information  to  maintenance  men,  and  it  is 
Important  to  know  how,  or  more  precisely  where,  this  information 
is  conveyed.  Of  these  two  findings,  the  first  justifies  study 
of  the  problem  of  efficient  failure  reporting.  The  second 
finding  directs  attention  toward  fruitful  areas  where  the 
efficiency  of  failure  reporting  can  be  Improved.  (Both  of  these 
Issues  were  treated  in  paragraph  1 of  this  section). 

It  is  equally  Important  to  know  that  symptoms  convey 
information  to  those  outside  the  maintenance  environment: 
research  workers,  project  officers,  base  commanders,  or  Air  Force 
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systems  personnel.  In  the  previous  paragraph  the  question  was, 

"How  can  we  predict  black  box  removal  rates  from  knowledge  of  the 
reported  symptom?"  Here  the  question  Is,  "What  else  can  we 
predict?"  Since  this  Is  the  case,  the  following  Information  Is 
organized  around  dependent  variables  rather  than  around  the  In- 
dependent variables  as  done  previously. 

2.1  The  Approach 

The  approach  followed  was  almost  what  statisticians  refer 
to  as  "fishing".  A dependent  variable  (c.f.  Section  III, 
paragraph  2)  was  selected,  e.g.  malfunction  verification,  and  It 
was  explored  for  relationship  to  any  Independent  variable  (a) 
available  In  the  data,  and  (b)  reasonable  In  light  of  what  Is 
known  or  assumed  about  maintenance  behavior. 

In  general,  this  meant  that  a large  number  of  analyses  were 
performed  which  produced  nonsignificant  results.  These  particular 
analyses  are  not  reported  here  because  they  would  obscure  the 
presentation  of  the  more  positive  findings.  However,  the 
observation  of  nonsignificant  results  Is  not  considered  trivial, 
and  these  analyses  are  not  to  be  Ignored.  Appendix  VI  lists  all 
the  analyses  conducted  In  the  present  study.  The  details  are 
not  given,  but  the  list  Indicates  what  sorts  of  relationships 
were  considered.  (The  reader  may  not  need  to  be  reminded  that 
failure  to  observe  significance  Is  only  partially  correlated  to 
absence  of  effect.) 

2.2  Maintenance  Time 

Maintenance  time  can  be  separated  into  a number  of  mutually 
exclusive  categories  of  activity.  In  the  present  study  seven 
such  categories  were  used  (see  Section  III,  paragraph  2).  (Six 
were  available  for  the  ASB/4  data  and  one,  total  time,  was 
available  in  the  F-lo6  flight  line  data.) 

Neither  the  data  available  nor  the  scope  of  this  study 
allowed  the  application  of  a strong  experimental  design  to  the 
analysis.  Maintenance  time  could  be  compared  to  only  one  aspect 
of  a symptom  at  a time,  but  within  that  aspect  the  comparison 
was  statistically  sound.  A simple  randomized  design  analysis 
of  variance  was  employed.  (This  allows  the  simultaneous 

comparison  on  one  variable  of  groups  of  data  of 

different  sizes.) 

Table  9 lists  the  significance  levels  of  the  various  relation- 
ships detected.  In  this  table  the  categories  of  symptoms  labeled 
Mood,  Description,  and  Intemittent  are  defined  as  before.  "Repair 
Job"  and  "Check  Job"  refer  to  carefully  defined  sub Ject -description 
pairs  of  frequently  occurring  (n>10)  corrective  and  preventive 
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maintenance  requests.  Table  10  contains  some  examples  of  how 
this  was  done.  (Unfortunately,  the  computer  program  had  not 
been  completely  debugged  on  the  first  attempt  to  study  the  effect 
of  the  Subject  of  the  s3miptom  and  there  was  no  opportunity  to 
repeat  the  analysis.) 


TABLE  9 

LEVELS  OP  SIONIPICANCE  FOR 
SYMPTOM  CHARACTERISTICS  VERSUS 
CATEGORIES  OP  MAINTENANCE  TIMES 
A SB/4  DATA 


Maintenance  Time 
M.  V.  F.  L.  P.  P.  Rep. 

.05 

.005 

.05  .05  .05  .05 

.05  .001  .001  .001 

.01 

An  empty  cell  denotes  a nonsignificant  result 


In  the  P-106  data,  where  symptom  is  coded  more  generally, 
total  system  down  time  is  correlated  (see  Section  IV,  paragraph 
3 ) to  symptom.  The  correlation  Implies  that  37^  of  the  variability 
in  total  system  down  time  can  be  accounted  for  by  the  symptom 
per  se. 

2 . 3 Malfunction  Verification 

In  an  effort  to  isolate  the  variables  affecting  the 
probability  of  malfunction  verification,  flight  line  ASB/4 
symptoms  were  separated  on  the  basis  of  their  observed  likeli- 
hood of  verification.  All  operator  complaints  were  classified 
into  what  seemed  to  be  distinct  Jobs  (see  Table  lO).  The 
probability  of  malfunction  verification  was  calculated  for  each 
Job,  and  each  probability  was  then  converted  into  a Z score  with 
respect  to  the  distribution  of  all  such  probabilities.  Looking 
only  at  those  Jobs  more  than  one  standard  deviation  (high  and 
low)  away  from  the  mean  probability  of  verification,  an  attempt 
was  made  to  isolate  the  factors  determining  probability  of 
malfunction  verification.  A very  careful  examination  of  these 


M.  Ad 


Symptom 


Prep. 

Mood 

.05 

Description 

Repair  Job 

Check  Job 

.01 

Intermittant 

.01 
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TABLE  10 


AN  EXAMPLE  OF  JOB  DEFINITION 


r ^ ^ 


high  and  low  verification  symptoms  Including  a column  by 

column  comparison  of  the  data  cards  failed  to  yield  any 

new*  clues  as  to  the  causes  of  verification  (or  failure  to 
verify) . 

A part  of  a consideration  of  the  problem  of  malfunction 
verification  concerns  what  happens  to  units  after  they  are 

removed  the  bench  check  results.  An  examination  of  407 

F-I06  shop  symptoms  suggests  that  the  factors  Influencing  "bench 
check  OK"  rates  are  much  easier  to  isolate  than  those  affecting 
verification.  Symptoms  described  to  the  shop  by  the  flight  line 
maintenance  crew  which  "bench  check  OK": 

(1)  Are  more  likely  to  contain  the  word  "intermittent" 
(p.<.Ol)** 

(2)  Are  more  likely  to  contain  a qualifying  phrase 
(P<.01) 

(3)  Contain  more  words  than  "bench  check  bad"  symptoms 
(P  <.oi)t 

(4)  Contain  more  phrases  than  "bench  check  bad"  symptoms 

(p  <.io) 

In  the  light  of  these  results,  the  occurrence  of  the  word 
"Intermittent"  in  a flight  line  symptom  has  a curious  effect. 

The  probability  of  unit  removal  (that  is  the  probability  of 
reported  verification  and  unit  action)  is  98^  as  compared  with 
62^  for  similar  symptoms  without  a word  Intermittent.  This 
difference,  perhaps  a reflection  of  maintenance  behavior  more 
than  of  differences  in  the  probability  of  malfunction  verification, 
is  significant  beyond  the  .001  level. tt 


* In  1961,  analysis  of  all  of  the  ASB/4  data  in  connection  with 
the  contract  for  which  it  was  collected,  showed  that  the  proba- 
bility of  verification  decreased  with  (a)  Increased  initial 
delay  time  (b)  increased  malfunction  number  within  a work  order 
and  (c)  increased  numbers  of  unverified  malfunctions  within  a 
work  order.  In  Table  7 mention  was  made  of  the  effect  of 
symptom  length  upon  verification.  And  in  Section  IV,  paragraph 
1.9  the  discussion  of  the  relationship  between  intermittancy 
and  verification  was  begun. 

**  Significance  levels  using  a binomial  probability  test. 

t Significance  levels  using  a chi  square  statistic, 
tt  Significance  level  using  a binomial  probability  test. 


2.4  Personnel  Assignment 


For  these  data,  there  are  three  methods  of  detecting  per- 
sonnel assignment  biases.  First,  In  the  ASB/4  data  manning 
profiles  can  be  compared  (numbers  of  men  at  various  skill  levels) 
for  various  units  removed.  This  treats  "unit  removed"  as  a 
definition  of  the  Job.  The  relationship  between  manning  profile 
and  unit  removed  Is  significant  (p  <.00l).  This  simply  means 
that  certain  units  are  removed  by  certain  numbers  of  men  or 
certain  skill-levels  or  both. 

Second,  one  can  separate  ASB/4  symptoms  themselves  Into  Job 
descriptions.  Jobs  are  described  as  before  (see  Table  lO) . The 
relationships  between  numbers  of  men  at  various  skill  levels  are 
significant  for  both  preventive  and  corrective  maintenance 
(p  <.01  for  3 and  7 level  technicians  and  p <.05  for  5 level). 

The  differences  between  preventive  and  corrective  maintenance 
are  also  significant  (p  <.0l). 

Third,  for  the  Dover  F-106  data  It  Is  possible  to  "correlate" 
(see  Section  4.3)  symptom  to  the  Identity  of  the  maintenance  man 
assigned.  This  yields  a constraint  of  2d. 6^. 

3,  INFORMA LYTICS* 

During  the  progress  of  this  research,  the  Investigators  were 
requested  to  exercise  a portion  of  their  collective  acumen  on  a 
potpourri  of  F-106  data  which  had  been  collected  In  a large  study 
directed  at  defining  system  problems  rather  than  at  solving 
specific  problems  (ref.  ?)•  It  was  reasoned  that  the  Information 
theory  metric  measures  the  amount  of  Information  communicated 
through  a channel.  An  Input  device  codes  the  Information;  a 
transmission  line  carries  the  Information;  an  output  device  decodes 
the  Information.  This  Is  the  general  problem  description.  It  Is 
a GENERAL  problem  description  and  Information  theory  MEASURES 
the  amount  of  Information  transmitted.  Any  pair  of  cooccurring 
events  can  be  cast  In  this  problem  description. 

Informalytlcs  Is  a quantification  technique  which  can  be 
applied  to  qualitative  data. 


* Informalytlcs  Is  a word  coined  here  to  refer  to  the  use  of 
Information  theory  as  an  analytic  tool  to  evaluate  data.  As 
such  It  Includes  the  application  of  Information  and  uncertainty 
measures  as  a technique  for  assessing  correlation,  dependence, 
and  structure,  and  It  also  Includes  the  statistics  necessary  to 
evaluate  the  likelihood  Implications  of  any  particular  obser- 
vation, (cf  Section  3.1). 
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As  this  concept  matured,  the  work  required  for  the  two 
studies  (the  study  reported  here  and  the  one  on  the  F-IO6) 
merged.  Each  study  profited  from  the  activity  of  the  other. 

The  F-I06  study  exploited  the  computational  philosophy  developed 

for  this  study  and  funded  the  computerization  of  the  second 

set  of  programs,  the  bivariate  analyses.  The  present  study 
obtained  a good  sample  of  shop  maintenance  data  and  achieved  a 
much  enlarged  point  of  view. 

The  following  paragraphs  repeat  the  significant  content 
of  the  F-I06  study.  This  is  appropriate  because  (a)  the  F-IO6 
program  was  directed  at  an  audience  unrelated  to  this  one  (b) 
this  portion  of  the  F-I06  work  is  corroborative  to  other  portions 
of  the  present  study  (c)  this  work  represents  an  extension  of 
concepts  first  applied  in  the  present  study  and  as  such  (d)  this 
work  Introduces  a new  tool  to  the  technology  of  quantifying  human 
performance.  The  arrangement  of  these  paragraphs  is  similar  to 
that  employed  in  section  IV,  paragraph  2,  one  subsection  for  each 
maintenance  or  system  criterion. 

3 . 1 Malfunction  Verification 

The  total  uncertainty  in  reported  verification  is  1.08^96 
bits.  Table  11  summarizes  the  results  of  several  correlations 
of  this  event  with  other  attributes  of  the  maintenance  environ- 
ment. The  first  eight  of  these  correlations  are  with  potential 
predictor  variables,  the  second  four  relate  to  events  that  occur 
after  the  verification  event.  These  are  included  to  gain  a picture 
of  how  well  verification,  as  a measure  of  maintenance  efficiency, 
is  related  to  other  measures  of  maintenance  efficiency. 

Table  11,  and  the  others  like  it  in  succeeding  paragraphs 
of  this  section,  presents  the  results  of  information  theory 
analysis  of  the  variables  listed.  The  first  column  Indicates 
how  many  variables  were  used  in  the  correlation  to  the  major 
variable;  in  this  table  the  major  variable  is  malfunction 
verification.  The  second  column  names  the  variables  used.  The 
third  column  indicates  how  much  uncertainty  is  available  in  the 
predictor  variable.  (The  amount  of  information  gained  concerning 
the  predicted  variable  can  not  exceed  the  amount  available  in 
the  predictor  variable.) 

The  fourth  column  lists  how  much  predicted  uncertainty  is 
reduced  by  knowledge  of  the  predictor  variable. 

The  amount  of  uncertainty  predicted  by  the  predictor  variable, 
expressed  as  a precentage  of  the  predictor  variable, is  a measure 
of  the  efficiency*  of  the  prediction  (column  5).  If  all  the  data 


* See  Section  III,  paragraph  1.2. 
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TABLE  11 


PREDICTION  OF  MALFUNCTION  VERIFICATION 
U = 1.08496;  E = 10.0% 


Number 

of 

Variable 

Predictor 

Variable 

Uncertainty 

Predicted 

Effi- 

ciency, 

% 

Percent 
of  Total 
Accounted 
For 

1 

Pilot 

6.10§30 

.55439 

9.1 

51.1 

1 

Mission  Success 

1 . 66678 

. 39025 

23.4 

36.0 

1 

Symptom 

5.29164 

.32461 

6,1 

29.9 

1 

Aircraft  Number 

5.63709 

.26491 

4.7 

24.4 

1 

Maintenance  Man 

2.25079 

.14107 

6.3 

13.0 

1 

Maintenance 

Concept 

1 . 33009 

.13573 

10.2 

12.5 

1 

Base 

.88014 

.07846 

8.9 

7.2 

1 

Severity 

1 . 35768 

.04234 

3.1 

3.9 

1 

Action  Taken 

1.83494 

. 33660 

18.3 

31.0 

„ 1 

Method  of 

Troubleshootii 

2.78509 

.26264 

9.4 

24.2 

1 

Next-Flight 

Success 

2.03774 

.17996 

8.8 

16.6 

1 

Downtime 

3.48140 

.15175 

4.4 

14.0 

Maintenance  Con- 

- 

2 

cept  & Method 
of  Trouble- 
shooting 

3.56814 

.29092 

8.2 

26.8 

2 

Symptom  and  Pile 

3t  8.97451 

.97705 

10.9 

90.1 

1 

Mission  Success 

1.66678 

.39025 

23.4 

36.0 

2 

Mission  Success 
and  Pilot 

7.26773 

.74823 

10.3 

69.0 

3 

Mission  Success 
Pilot  & Symptom 

9 . 15490 

1.02601 

11.2 

94.6 

Note;  "u"  Is  the  uncertainty  and  "E"  Is  the  efficiency  (if 
all  data  were  used) 

Instructions  for  reading  the  table  follow;  those  for 
calculating  Its  entries  were  provided  in  Section  III, 
paragraph  1.1,2;  those  for  assessing  the  statistical 
signlflceuice  of  those  entries  were  provided  In 
Section  III,  paragraph  2.  
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were  used  to  predict  verification,  efficiency  would  be  as  low  as 
10.0^.  An  appropriate  value  of  this  sort  Is  presented  at  the  top 
of  each  table  In  order  to  assess  the  relative  usefulness  of  any 
of  the  tabled  predictions. 

The  predicted  uncertainty  reduction  divided  by  the  total 
uncertainty  of  the  predicted  variable  Is  the  percentage  of  the 
total  uncertainty  which  can  be  obtained  through  knowledge  of  the 
predictor  variable.  This  percentage  can  be  used  directly  as 
a measure  of  correlation.  Furthermore,  It  Is  mathematically 
permissible  to  estimate  an  analogous  product  moment  correlation 
coefficient  as  a function  of  the  square  root  of  this  percentage 
(as  explained  in  Section  III,  paragraph  1.1.2  and  ref.  5). 

Finally,  In  the  lower  portion  of  each  table  the  terms  of  a multi - 
variable  prediction  scheme  have  been  collected.  This  analysis 
is  similar  to  multiple  regression.  Here,  however,  instead  of 
predicting  with  a formula,  one  predicts  with  a table. 

3 . 2 Next  Flight  Success 

Next  flight  success  Is  an  evaluation  of  whether  or  not  the 
symptoms  on  this  flight  were  repeated  on  the  next  flight.  As 
such  it  Is  a clear  measure  of  maintenance  efficiency,  but  Its 
Interpretation  must  be  tempered  because  the  symptom  could  have 
repeated  spontaneously,  that  Is,  it  could  be  another  malfunction 
of  the  same  subsystem. 

If  maintenance  personnel  were  perfect,  the  probability  of 
next  flight  success  would  be  independent  of: 

(1)  Symptom  - because  all  malfunctions  would  be  fixed. 

(2)  Verification  - because  all  malfunctions  would  be 
fixed  and  all  unverified  malfunctions  would  be  good 
systems . 

(3)  Maintenance  Men  - because  all  men  would  be  equally 
perfect . 

(4)  Base  - because  both  bases  would  be  equally  perfect. 

Table  12  checks  some  of  these  predictions  and  several  others. 
The  table  is  to  be  interpreted  as  before. 

3 . 3 Down  Time 

Down  time  is  the  most  common  measure  of  maintainability. 

Table  13  examines  this  measure  as  a function  of  six  other 
variables.  For  the  upper  portion  of  the  table  down  time  was 
considered  in  hours  only,  disregarding  tenths  of  an  hour.  In 
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TABLE  12 


PREDICTION  OF  NEXT-PLIGHT  SUCCESS 
U = 2.03774;  E = 18. 8?^ 


Number 

of 

Variable 

Predictor 

Variable 

Uncertainty 

Predicted 

Effi- 

ciency, 

Percent 
of  Total 
Accounted 
For 

1 

Pilot 

6.10230 

.55925 

9.1 

28.5 

1 

Aircraft  Number 

5.63709 

.46848 

8.3 

23.0 

1 

Symptom 

5.29164 

.40293 

7.6 

19.8 

1 

Method  of  Trouble- 
shooting 

2.78509 

.27952 

10.0 

13.7 

1 

Downtime ♦ 

3.48140 

.27801 

8.0 

13.6 

1 

Unit 

4.86252 

.23669 

4.9 

11.6 

1 

Maintenance  Man** 

2.25079 

.21129 

9.4 

10.4 

1 

Verlflcat Ion 

1.08496 

. 17996 

16 . 6 

8.8 

1 

Maintenance  Con- 
cept 

1 . 33009 

. 16624 

12.5 

8.2 

1 

How  Mai 

2 .47726 

.13694 

5.5 

6.7 

2 

Symptom  and  Unit 

7.98804 

.85442 

10.7 

41.9 

Method  of  Trouble- 

. 

2 

shooting  & Main- 
tenance Concept 

- 3.56814 

. 37202 

10.4 

18.3 

Action  Taken 

3 

Method  of  Trouble- 
shooting 

[ 4.70519 

. 50382 

10.7 

24.7 

Maintenance  Concept 

1 

Pilot 

6.10230 

.55925 

9.1 

28.5 

2 

Pilot  and  Symptom 

8.97189 

1 . 64989 

18.4 

81.1 

Pilot  and  Symptom 
and  Unit 

10.40586 

1.78934 

17.2 

88.0 

♦Hours  only 

♦♦Both  bases,  thus  data  have  one  number  for  all  Selfrldge  men 
and  one  for  each  Dover  man. 

Note:  Instructions  for  reading  the  table  follow;  those  for 
calculating  Its  entries  were  provided  In  Section  III, 
paragraph  1.1.2;  those  for  assessing  the  statistical 
significance  of  those  entries  were  provided  In  Section  III, 
paragraph  2 . 
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TABLE  13 


PREDICTION  OF  DOWNTIME 
U = 5.^9548*;  E = 50.85^ 


Number 

of 

Variable 

Predictor 

Variable 

Uncertainty 

Predicted 

Effi- 

ciency, 

Percent 
of  Total 
Accounted 
For  ' 

1 

Pilot 

6.09102 

1.21223 

19.9 

35.1* 

1 

Symptom 

5.29164 

.97879 

18.5 

28.3* 

1 

Unit 

4 . 84940 

.62937 

13.0 

18.2* 

1 

Method  of  Trouble- 
shooting 

2.74523 

.54021 

19.7 

15.6* 

1 

Maintenance  Concept 

1 . 33009 

.24159 

18.2 

7.0* 

1 

When  Discovered 

1.18339 

.22320 

18.9 

6.5* 

1 

Symptom 

5.29164 

2.06049 

38.9 

37.5 

2 

Symptom  and  Unit 

7.97015 

3.24639 

40.7 

59.1 

3 

Symptom  and  Unit 
and  Trouble- 
shooting Method 

8.40605 

3.65380 

43.5 

66.5 

4 

Symptom  and  Unit 
and  Trouble- 
shooting Method 
and  Maintenance 
Concept 

8 . 54002 

3.78066 

44.3 

68.8 

♦Considered  as  a three-digit  code.  Single  variable  comparisons 
used  downtime  as  a two-digit  code  (hours),  U = 3.^5271 


Note:  Instructions  for  reading  the  table  follow;  those  for 
calculating  its  entries  were  provided  In  Section  III, 
paragraph  1.1.2;  those  for  assessing  the  statistical 
significance  of  those  entries  were  provided  In  Section  III, 
paragraph  2 . 
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view  of  the  success  of  the  single  variable  predictions,  tenths 
of  an  hour  were  Included  in t he  multivariable  analysis. 

3.4  Unit 

It  is  a stretch  of  the  imagination  to  suggest  that  the  unit 
pulled  in  a maintenance  action  is  somehow  a measure  of  the 
efficiency  of  that  action.  The  only  aspects  of  this  which  directly 
hint  of  efficiency  are  those  associated  with  whether  or  not  the 
unit  is  really  bad.  These  aspects  will  be  discussed  in  the  next 
two  sections.  Here,  concern  is  only  with  discovering  those 
aspects  of  the  total  maintenance  situation  which  would  help  to 
predict  which  units  will  be  involved.  (That,  after  all. is  the 
problem  of  information  transmission  to  maintenance  men.)  The 
progress  of  this  search  is  documented  in  Table  l4. 

3 . ^ A Summary 

The  reader  will  notice  that  Informalytics  is  a dramatic 
method  for  pinpointing  problem  areas  and  for  assessing  the 
relative  Impact  of  selected  variables  upon  system  performance. 

In  the  F-106  avionics  systems  it  appears  (based  upon  the  sizes 
of  the  correlations  available  in  Tables-  11,  12,  13,  and  l4)  that 
variability  in  system  performance  is  2:1  a personnel  problem 
rather  than  an  equipment  problem. 

It  was  not  possible  to  check  this  observation  by  comparing 
it  to  analyses  in  the  ASB/4  data,  as  those  data  did  not  Include 
personnel  identity  in  a sufficient  number  of  cases. 


TABLE  14 


PREDICTION  OF  UNIT 
U = 4.86252;  E = 44. 95^ 


Number 

of 

Variable 

- 

Predictor 

Variable 

Uncertainty 

Predicted 

Effi- 

ciency, 

Percent 
of  Total 
Accounted 
For 

1 

Symptom 

5.29164 

2.16612 

40.9 

44.5 

1 

Aircraft  Number 

5.63709 

1.12130 

19.9 

23.1 

1 

Pilot 

6.09102 

.28295 

4.6 

5.8 

1 

When  Discovered 

1 . 18339 

.25205 

21.3 

5.2 

1 

How  Mai 

2.47726 

.94531 

38.2 

19.4 

1 

Downtime 

3.45261 

.62937 

18.2 

12.9 

1 

Next-Flight  Success 

2.03774 

.23669 

11.6 

4.9 

1 

Symptom 

5.29164 

2.16612 

40.9 

44.5 

2 

Symptom  and  Pilot 
Symptom  and  Pilot 

8.94240 

3.37343 

37.7 

69.4 

3 

and  Trouble- 
shooting Method 

9.12336 

3.48108 

38.2 

71.6 

Note:  Instructions  for  reading  the  table  follow;  those  for 
calculating  its  entries  were  provided  in  Section  III, 
paragraph  1.1.2;  those  for  assessing  the  statistical 
significance  of  those  entries  were  provided  in  Section  III, 
paragraph  2. 
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SECTION  V 


DISCUSSION  AND  CONCLUSIONS 

Section  IV  provided  a quick  overview  of  the  major  analyses 
associated  with  this  study,  but  did  not  suggest  many  interpre- 
tations of  those  analyses.  Rather,  in  the  Interests  of  uncom- 
plicated presentation.  Section  IV  primarily  presented  the  results; 
this  section  will  consider  some  of  the  reasonable  interpretations. 

1 . INFORMATION  TRANSMISSION 

The  results  of  this  study  clearly  demonstrate  that  symptoms 
carry  information;  it  is  the  purpose  of  this  section  of  the 
report  to  Interpret  where  and  how  this  information  is  conveyed. 

1.1  Unit  Designation 

There  can  be  no  doubt  that  the  reported  symptom  conveys 
information  of  assistance  in  trouble-shooting.  In  fact,  it 
is  clear  (a)  that  there  is  more  information  available  than 
maintenance  men  use  (since  they  do  not  trouble-shoot  optimally, 
see  section  IV  paragraph  1.2)  (b)  that  there  is  more  Infonnatlon 
available  to  the  operator  than  the  symptom  contains,  (since  the 
operator  himself  is  a significant  variable  in  the  selection 
of  the  symptom)  (see  section  III,  paragraph  1.1.2  and  paragraph 

2.2  of  this  section) and  (c)  there  is  more  information  available 
to  the  maintenance  man  than  is  contained  in  the  symptom  (since 
he  is  more  efficient  at  trouble-shooting  than  a computer  could 
be  using  the  same  symptoms  and  an  optimum  search  strategy  but 
no  additional  Information). 

The  subject  of  the  reported  symptom  was  again  shown  to 
contain  most  of  the  unit  designation  information.  (It  reduces 
total  uncertainty  as  to  which  black  box  is  the  bad  one  by  more 
than  60%.)  The  other  characteristics  of  the  malfunction  report 
seem  to  constrain  malfiinction  verification  likelihood  and  mal- 
function verification  time. 

1.2  Malfunction  Verification 


The  symptom  conveys  a considerable  amount  of  information 
concerning  the  issue  of  malfunction  verification,  but  most  of 
it  is  about  personnel  behavior  and  not  about  the  true  verifi- 
cation of  malfunctions. 

Malfunction  verification  seems  to  be  a paperwork  problem. 
Personnel  are  motivated,  for  a variety  of  reasons,  to  claim 
that  a malfunction  has  been  verified.  (If  a maintenance  man 
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falls  to  verify  a malfunction  and  the  symptom  Is  reported  again 
on  the  following  flight,  he  suffers  more  than  chagrin.)  Accord- 
ingly, It  Is  not  surprising  that  the  factors  which  produce  high 
flight  line  verification  rates  also  tend  to  produce  high  bench 
check  OK  rates. 

An  examination  of  the  407  F-IO6  shop  symptoms  (written 
by  the  maintenance  man  to  the  shop ) showed  that  Intermittent 
symptoms  have  a much  higher  "bench  check  OK"  rate  than  stable 
symptoms.  This  Information,  and  the  fact  that  Intermittent 
symptoms  are  much  more  likely  to  lead  to  a unit  removal  than 
stable  symptoms  (98^  of  the  time  vs.  62^  of  the  time)  lead  to 
some  Interesting  conclusions.  It  appears  that  the  maintenance 
man  has  less  faith  In  his  failure  to  verify  an  Intermittent 
symptom  than  he  does  In  his  failure  to  verify  a stable  symptom. 
This  Is  very  much  like  the  partial  reinforcement  effect  In 
psychology.  The  maintenance  man  may  test  the  equipment  and 
find  nothing  wrong,  but  since  the  symptom  was  Intermittent 
anyway  he  may  decide  that  It  "just  wasn't  malfunctioning  on 
this  try"  and  go  ahead  and  replace  a suspected  unit  - often 
removing  a good  one.  (This  Is  not  meant  to  Imply  that  Inter- 
mittent symptoms  do  not  exist,  but  merely  that  they  are  harder 
to  verify.  — note  that  verification  time,  Indeed  all  categories 
of  time  are  longer  for  Intermittent  symptoms). 

In  connection  with  Intermittent  symptoms  It  Is  Interesting 
to  note  that  In  spite  of  a near  100^  verification  rate,  they 
contain  a disproportionate  number  of  misleading  reports.  Four- 
teen symptoms  were  coded  by  the  field  engineers  as  "misleading" 
symptoms.  Of  these  misleading  symptoms,  55^  contained  the  word 
"intermittent,"  whereas  only  of  all  of  the  reported  symptoms 
contained  this  word. 

The  presence  of  a modifier  or  qualifying  phrase  In  F-IO6 
shop  symptoms  Is  positively  related  to  high  "bench  check  OK" 
rates.  This  Is  probably  true  for  much  the  same  reason  as  It 
Is  for  Intermittent  symptoms.  A modifier  often  Implies  that  a 
symptom  occurs  only  at  certain  times  (e.g.  "in  banks  and  turns" 
or  "after  5 miles")  and  Is  therefore  quite  hard  to  verify  on 
the  flight  line,  where  "banks  and  turns"  are  difficult  (If 
not  Impossible)  to  simulate.  For  this  reason,  again,  good 
units  may  be  removed  If  they  are  suspected  of  causing  the  diffi- 
culty. 

On  the  other  hand,  for  flight  line  symptoms,  the  presence 
of  a modifier  corresponds  to  an  Increased  likelihood  of  veri- 
fication. It  Is  not  certain  whether  this  Is  due  to  an  Increased 
likelihood  that  the  malfunction  really  occurred,  or  due  to  the 
effect  Increased  verbage  has  on  the  credibility  of  the  report. 
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1 .3  Maintenance  Time 

The  symptom  Is  the  first  element  of  the  maintenance  event 
and  may  be  legitimately  assumed  to  be  a significant  determiner 
of  subsequent  action.  After  all,  there  would  be  no  Justification 
for  reporting  a symptom,  nor  for  listening  to  It,  If  one  didn't 
believe  that  It  reduced  his  uncertainty  about  what  went  wrong. 
Knowing  what  Is  wrong  should  be  tantamount  to  knowing  how  long 
It  will  take  to  fix  It.  The  data  here  suggest  that  maintenance 
time  could  be  predicted  from  symptoms. 

When  Jobs  are  carefully  defined  In  terms  of  the  reported 
symptoms,  every  category  of  maintenance  time  seems  to  be  a 
function  of  the  Job — even  administrative  delay  time  I This 
result  Is  Intuitively  reasonable  and  an  "Interpretation"  Is 
obvious  as  stated. 

Preparation  time  and  fault  location  time  are  significantly 
affected  by  the  mood  of  the  symptom.  The  trends  here  are  for 
Increasing  objectivity  of  mood  to  correspond  with  decreasing 
time  requirements  for  trouble  shooting.  This  Is  consistent 
with  an  Interpretation  which  suggests  that  as  operators  become 
less  "emotionally  Involved"  In  reporting  the  malfunctions,  they 
become  more  efficient  In  really  communicating  the  difficulty. 
Notice  that  the  differences  in  Information  transmission  were 
also  consistent  with  this  view.  (Increased  "emotionality" 
could  correspond  to  an  Increased  likelihood  of  an  Invented 
symptom,  but  then  the  trend  In  probability  of  verification  would 
be  exactly  opposite  to  that  observed).  It  might  be  appropriate 
to  reflect  on  the  problem  of  mood  and  consider  training  and 
evaluation  and  morale  procedures  which,  at  little  expense, 
could  move  operator  Involvement  to  a more  efficient  point  on 
the  motivation-performance  curve  (shades  of  the  Yerkes-Dodson 
law) . 


Malfunction  verification  time  Is  significantly  related 
to  the  description  In  the  symptom.  This  Is  probably  because 
the  description  describes  what  the  system  Is  or  is  not  doing 
and  it  Is  therefore  related  to  the  verification  criteria. 

Finally,  every  category  of  maintenance  time  Increases  for 
intermittent  symptoms  — probably  because  they  are  more  difficult. 
(This  Interpretation  Is  also  supported  by  the  higher  incidence 
of  high  level  technicians  on  "Intermittent"  Jobs.) 

2.  THE  MAINTENANCE  SYSTEM 

In  section  IV,  paragraph  3^  Informalytics  (Information 
theory  used  as  an  analytic  tool)  Indicated  a number  of  useful 
relationships  between  various  characteristics  of  the  maintenance 


environment  and  certain  measures  of  system  performance.  These 
relationships  are  collected  and  Interpreted  in  the  following 
paragraphs. 

2 . 1 Organizational  Structure 

The  two  F-106  bases  studied  are  differentially  effective 
at  maintenance.  One  has  a probability  of  next  flight  success 
of  .70  while  the  other  has  a probability  of  next  flight  success 
of  .49.  These  values  are  statistically  different,  but  that  is 
not  as  important  as  the  fact  that  they  are  functionally  dif- 
ferent. The  first  gets  more  planes  flying  successfully.  There 
are  a lot  of  differences  between  the  two  bases  in  terms  of 
flight  line  procedures.  Maintenance  concepts  (item  21 
Appendix  II ) are  differentially  applied  to  differentially 
likely  symptoms.  For  example,  at  the  first  base,  MA-1  personnel 
are  more  frequently  used  by  themselves  than  they  are  at  the 
second  base.  However,  the  most  important  organizational  factor 
seems  to  be  the  assignment  of  men  and  aircraft  to  squadrons. 

At  the  first  base  there  is  both  "pride  of  ownership"  and  a 
"feeling  of  responsibility"  for  the  aircraft.  At  the  second 
base,  the  complexion  of  the  maintenance  Job  is  different. 

Here  a technician  is  really  confronted  with  an  assignment  to 
dispose  of  a maintenance  action  on  a relatively  anonymous  air- 
frame, while  at  the  first  base  he  must  fix  one  of  his  aircraft. 
It  appears  from  the  data  that  such  a difference  in  attitude 
might  have  a demonstrable  effect  on  maintenance  performance. 

2.2  Pilot 


To  begin  with,  very  late  in  the  F-IO6  analysis  it  was 
learned  that  pilot  number  is  not  unique;  halfway  through  the 
data  collection  period  many  pllo''‘s  exchanged  code  numbers. 

This  charge  means  that  pilot  niomber  XX  now  refers  to  a parti- 
cular pair  of  pilots  rather  than  to  a specific  pilot  indivi- 
dually.* 

Using  the  information  theory  approach  (see  section  III, 
paragraph  1.1.2),  pilot  is  correlated  to  symptom;  about  36^ 
of  the  uncertainty  in  symptom  can  be  obtained  from  the  pilot's 
code  number.  Since  aircraft  number  is  also  related  to  symptom, 
it  is  possible  that  the  pilot-symptom  correlation  is  spurious 
and  really  due  to  a pllot-alrcraft-symptom  correlation.  The 
correlation  between  pilot  and  aircraft  within  a given  squadron 
for  one  whole  year  was  checked  — the  effect  is  small  (overlap 
of  only  11^).  Apparently,  pilot  is  related  to  symptom.  It 
is  not  possible,  in  this  sample  size,  to  determine  if  this 


*Thls  confounding  means  that  the  calculated  "correlations"  with 
pilot  are  probably  lower  than  they  should  be.  The  time  required 
to  redo  the  analysis  exceeded  the  time  left  for  completing  the 
study. 


relationship  exists  because  some  pilots  report  only  certain 
symptoms,  or  because  some  pilots  report  at  random,  or  both, 
(further  study  would  be  required  to  answer  this  question.) 

Pilot  number  Is  significantly  related  to  down  time  (355^). 
This  Is  probably  a reflection  of  the  effect  noted  above.  A 
pilot  reports  a symptom  which  does  not  contain  a lot  of  Infor- 
mation (e.g.  he  makes  an  error  In  reporting  because  he  Is 
Inexperienced,  or  he  reports  a bogus  symptom  In  order  to  cover 
a mistake  he  made)  and  maintenance  time  Increases.  Another 
pilot  gets  right  to  the  point  with  his  symptom  and  maintenance 
time  decreases. 

Pilot  number  Is  correlated  to  next  flight  success,  but 
the  reasons  for  this  are  not  apparent.  Do  some  pilots  fall  to 
report  malfunctions?  Do  some  pilots  give  their  symptoms  so 
loosely  that  maintenance  personnel  can't  find  the  problem? 

Do  some  pilots  report  their  systems  so  concisely  that  maintenance 
personnel  can  easily  peak  the  equipment  up? 

Finally,  pilot  n\amber  Is  significantly  correlated  to 
malfunction  verification  (51^).  A full  one-half  of  the  uncer- 
tainty concerning  whether  or  not  a given  malfunction  will  be 
verified  can  be  obtained  If  one  knows  who  made  the  complaint, 

1 .e . even  If  you  Ignore  what  the  complaint  was. 

2 .3  The  Maintenance  Man 


For  the  ASB/4  data,  the  only  Indications  of  personnel 
assignment  bias  possible  were  dependences  of  skill  level  and 
numbers  of  personnel  upon  symptom  and  unit  action.  Chi 
square  tests  Justified  the  claim  that  such  a dependence  existed, 
but  It  could  have  been  as  a result  of  the  Job,  that  Is,  It 
could  have  occurred  after  work  started  and  not  necessarily  be 
an  assignment  bias.  (For  either  condition  the  relationship 
proves  that  symptom  per  se  Is  a commentary  on  Job  complexity.) 

For  the  F-106  data  at  Dover,  flight  line  personnel  codes 
were  available  for  the  personnel  assigned  to  the  maintenance 
action.  There  Is  an  assignment  bias;  symptom  and  man  correlate 
{26.6%).  At  least  a portion  of  this  Is  to  be  expected  because 
some  of  the  men  belong  to  the  Flight  Control  Shop  and  some 
belong  to  the  Radar  Shop.  The  overlap  between  man  1 and  man  2 
Is  33.2^  which  means  that  a very  strong  assignment  bias  Is 
operating  within  shops. 

Maintenance  man  code  number  overlaps  with  system  down  time 
29.7^.  A portion  of  this  could  be  due  to  a man- symptom-down 
time  chain  of  relationship.  It  Is  probably  not  all  due  to  that 
effect,  but  It  cannot  be  checked;  the  present  sample  size  Is 
too  small. 
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Maintenance  man  Is  also  related  to  trouble-shooting  method 
(25.7^).  To  the  extent  that  trouble-shooting  method  Is  Impor- 
tant, and  to  the  extent  that  this  relationship  Is  Independent 
of  symptom,  this  could  be  an  Important  observation.  Important 
choices  certainly  should  not  be  made  on  the  basis  of  Individual 
whim.  As  it  happens  however,  trouble  shooting  method  doesn't 
seem  to  be  particularly  Important  (see  below),  and  both  it 
and  man  depend,  at  least  a little,  upon  the  symptoms. 

The  story  with  maintenance  concepts  Is  different,  but  the 
result  Is  roughly  the  same.  The  overlap  Is  large  (21.6^) 
and  perhaps  spurious.  Certain  men  who  can  work  concept  2 
(FC  & M personnel  only)  cannot  work  concept  3 (MA-1  personnel 
only).  This  structure  could  be  enough  to  generate  the  observed 
bias . 

2.4  Trouble  Shooting  Strategy 

The  F-IO6  data  contained  a coded  description  of  the 
trouble-shooting  method  employed.  A correlation  of  this 
variable  produced  some  surprising  results.  To  be  sure,  symptom, 
the  maintenance  concept,  and  maintenance  man  are  each  importantly 
related  to  the  choice  of  a trouble-shooting  method.  On  the 
other  hand,  choice  of  trouble  shooting  method  does  not  appear 
to  be  very  importantly  related  to  the  various  measures  of  main- 
tenance efficiency.  In  particular  It  is  insignificantly  related 
to  the  probability  of  next  flight  success  and  unimpressively 
correlated  to  shop  action  taken. 

The  inability  to  detect  significant  relationships  does 
not  mean  that  the  choice  of  a trouble-shooting  method  Is 
unimportant.  As  was  noted  earlier,  the  problem  may  be  one  of 
applying  the  strategy  rather  than  of  choosing  it.  The  repaired 
vs.  bench  check-OK  record  for  all  units  involved  In  six  symptoms 
(RHH,  RH,  FN,  FT,  FA,  FD)  was  examined.  In  every  case  the 
total  number  of  bench  check-OK  units  was  less  than  the  minimum 
predicted  by  any  optimizing  strategy.  Clearly,  the  maintenance 
man  must  have  had  some  additional  information  ( cf  section  V, 
paragraph  l.l).  On  the  assumption  that  the  proportion  of 
bench  check-OK  units  would  be  a function  of  strategy,  Wilcoxon 
and  Chi  square  test  were  used  to  compare  the  observed  relative 
proportions  of  bench  check-OK' s to  the  various  predicted  pro- 
portions. All  of  the  optimum  strategies  were  easily  rejected. 

On  the  basis  of  this  analysis  it  appears  that  the  maintenance 
personnel  select  a trouble-shooting  strategy  on  the  basis  of 
what  they  think  will  work  in  that  context.  This  selection  of  a 
method  may  be  random  and  unimportant,  or  it  may  be  totally 
biased  but  assigned  on  a very  sound  basis.  At  any  rate,  the 
man  very  soon  gathers  a significant  amount  of  new  Information 
concerning  the  symptom,  and  this  change  in  the  maintenance 
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event  now  affects  his  subsequent  behavior  In  nonoptimum,  but 
presently  unde scrlbable, ways.  Thus  one  can  now  predict  how 
the  maintenance  man  will  start  trouble-shooting,  and  one  can  be 
sure  that  the  maintenance  man  Is  not  following  an  optimum  path, 
but  one  cannot  predict  what  happens  during  trouble-shooting, 
or  what  the  maintenance  man  will  end  up  doing.  One  can  predict 
the  unit  to  be  removed  as  a function  of  the  symptom,  but  one 
really  can't  predict  unit  very  much  as  a consequence  of  trouble- 
shooting procedure.  Based  on  trouble-shooting  strategy  one  can 
presently  predict  a trivial  amount  of  down  time,  and  this  only 
because  of  Inefficiencies  In  trouble-shooting  paths.  On  the 
other  hand  one  cannot  predict  next  flight  success  at  all, 
because  all  the  trouble-shooting  procedures  which  were  observed 
were  equally  effective  (or  equally  Ineffective  as  the  case  may 
be)  at  discoveri”g  the  offending  unit. 

In  general,  one  can  conclude:  (l)  the  maintenance  man  is 
operating  on  more  Information  than  that  contained  in  the  pilot's 
symptom.  (2)  the  maintenance  man  Is  not  using  any  optimum 

(A  decision  should  be  made  concerning  the  relative 
removals  and  tests  and  a prescribed  trouble-shooting 
procedure  should  be  required.)  (3)  the  average  F-IO6  total 
down  time  could  be  reduced  (perhaps  as  much  as  35  minutes  out 
of  --  minutes  currently  used)  If  maintenance  men  were  encouraged 
to  use  mock  up  checks  In  lieu  of  special  test  equipment,  local 
test  equipment  or  multimeter  testing.  Furthermore,  single  unit 
substitution  Is  a relatively  time  consuming  operation  but  whole- 
sale unit  swapping,  while  reducing  total  down  time  an  average 
of  1.65  hours,  encourages  high  bench  check -OK  rates.  This 
simply  underscores  the  fact  that  there  Is  a trade-off  here 
between  shop  and  flight  line  manhour  expenditures. 

2.5  Paper  Work 

In  the  data  analyzed  here  there  was  no  direct  way  In  which 
to  evaluate  the  Impact  of  paper  work  upon  measures  of  perfor- 
mance. The  absence  of  direct  paths  not  withstanding,  there 
Is  an  interesting  implication  available. 

It  appears  as  though  It  Is  possible  to  have  the  paperwork 
actually  degrade  system  performance  while  It  records  system 
performance  as  Improved.  Consider  an  operator  who  commits  an 
error  of  Judgement  on  one  of  his  attempted  missions.  He  Is 
evaluated  (paperwork)  on  his  performance,  so  he  blames  the 
equipment  In  order  to  get  "out  from  under"  the  error.  A main- 
tenance man  Is  sent  out  to  the  airplane,  and  he  gets  evaluated 
In  terms  of  how  many  malfunctions  he  says  he  fixes.  So  he 
"verifies"  the  complaint  and  pulls  a box  In  order  to  up  his 
"batting  average."  Meanwhile  the  aircraft  is  out  of  service. 
Finally  the  system  Is  placed  on  up  status  following  unnecessary 
repair.  It  now  has  a higher  probability  of  failing  on  the 


strategy, 
"costs"  of 
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next  flight  than  had  It  not  been  "repaired"  for  the  man  has 
disturbed  the  balance  of  the  system.  All  the  paperwork  says 
that  everybody  was  efficient,  but  the  system  as  a whole  is 
actually  worse  off. 

3 . RECOMMENDATIONS 

This  study  had  five  purposes  or  goals  (cf  Section  I, 
paragraph  2).  The  first  and  fourth  were  concerned  with  trying 
to  use  information  theory  on  maintenance  data,  and  the  exls- 
tence  of  Sections  IV,  paragraphs  1 and  3 demonstrates  the 
completion  of  those  tasks.  The  second  goal  was  concerned  with 
assessing  the  location  of  information  in  malfunction  reports, 
and  that  was  disposed  of  in  Section  IV,  paragraph  section  V, 

2 and  1.  The  remaining  two  goals  required  the  development  of 
recommendations  for  improving  maintenance  efficiency  and  for 
improving  system  efficiency.  These  tasks  are  dealt  with  in 
the  following  paragraphs. 

3.1  Training  Operators  in  Failure  Reporting 

It  is  clear  that  some  of  the  significant  characteristics 
of  a symptom  cannot  be  directly  converted  into  neat  little 
recipes  for  operator  training.  Training  an  operator  to 
always  say  "change  dessicators"  no  matter  what  is  wrong  certainly 
will  not  minimize  maintenance  time,  even  though  this  symptom  is 
a very  easily  diagnosed  symptom  and  the  job  referred  to  is  a 
very  easy  job.  The  problem  is  to  improve  the  transmission  of 
Information  already  available,  not  to  try  to  create  information 
to  be  reported.  (That  problem  will  be  attended  to  later.) 

It  is  recommended  that  an  on-site  training  program  be 
instituted  for  system  operators.  Such  program  should  run  for 
approximately  3 hours  and  should  cover: 

(a)  Maintenance  men  are  also  being  given  some 
refresher  training. 

(b)  Maintenance  cannot  be  effective  if  equipment 
is  not  referred  for  maintenance. 

( c ) Maintenance  cannot  be  effective  if  equipment 
malfxinctions  are  not  properly  described. 

(d)  Some  of  the  characteristics  of  the  data  of 
this  study. 


The  reasons  for  this  recommendation  are  several,  but 
major  support  can  be  found  in  these  three: 

1.  Some  pilots  appear  presently  to  be  withholding 
malfiinctlon  information  because  "What  the  hell,  they 
can't  fix  It  anyway." 

2.  Some  pilots  appear  to  be  writing  up  equipment 
malfunctions  too  glibly  and  not  giving  the  proper 
details. 

3.  Some  pilots  appear  to  be  writing  up  equipment 
malfunction  reports  for  their  own  errors. 

There  are  certain  things  which  can  be  done  outside  of 
the  area  of  training  operators  which  probably  would  have  broader 
and  more  Important  benefits. 

1.  Modify  the  debriefing  process,  based  upon  an 
analysis  of  the  equipment,  to  Include  prescribed 
questions  concerning  the  conditions  surrounding 
the  malfunction.  These  questions  would  be  used  to 
separate  one  symptom  from  another,  and  to  Indicate 
whether  or  not  a malfunction  has  really  occurred. 

(One  could  even  provide  the  operator  with  an  in-flight 
self-debrlefing  form). 

2.  Modify  the  operator  evaluation  system  so  that  he 

Is  evaluated  on  how  well  he  and  the  equipment  perform. 
This  will  remove  the  temptation  to  "blame"  the  equip- 
ment for  marginal  system  performance  which  Is  due 
to  the  operator,  or  to  both  the  operator  and  the 
equipment . 

3.  Modify  the  maintenance  evaluation  system  so  that 
the  maintenance  man  Is  also  evaluated  on  system 
performance.  If  this  Is  accomplished  successfully, 
say  by  using  conditional  probability  rules  and  by 
evaluating  teams  of  men  and  equipment,  then  personnel 
will  be  motivated  to  fix  equipment  rather  than  to 
"fix"  paperwork.  The  difference  should  be  Important. 

4.  Train  maintenance  men  to  understand  the  transmission 
of  Information  and  how  It  Implies  certain  rigid 
trouble-shooting  sequences.  This  should  improve 
their  efficiency  and  make  the  case  for  training 
operators,  above,  more  convincing  to  the  operators. 
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3 . 2 Trouble-Shooting 

The  data  here  are  entirely  consistent  with  the  theory 
and  predictions  of  the  Symptom  Matrix*  (ref.  l).  Clearly, 
the  symptom  does  carry  Information;  equally  clearly,  the 
maintenance  man  does  not  use  the  information  optimally.  This 
lack  of  optimality  may  be  because  (a)  he  doesn't  know  what  to 
do  with  the  information  he  receives,  (b)  he  is  not  able  to 
"perceive"  the  information,  (c)  he  does  not  accept  this  defini- 
tion of  optimum,  (d)  he  does  not  choose  to  do  his  best,  or  of 
course,  (e)  some  combination  of  these  effects. 

It  would  be  interesting  to  learn  which  of  these  situations 
prevails,  and  experiments  with  experienced  maintenance  men 
could  determine  that : 

1.  Prepare  a paper  and  pencil  test,  multiple  choice, 
asking  the  maintenance  personnel  which  of  "these" 
black  boxes  he  should  check  first  when  given  "this" 
symptom. 

2.  Construct  an  lllvunlnated  block  diagram  (like  a training 
aid)  and  ask  the  maintenance  man  to  predict  which 
black  box  will  be  bad  given  "this"  symptom.  Monitored 
over  a period  of  time,  this  probability  choice  experi- 
ment would  ascertain  (a)  the  course  of  development  of 
perception  of  a complex  probability  structure  and 

(b)  the  utility  structiire  of  the  maintenance  man's 
trouble-shooting  philosophy. 

3.  Collect  detailed  data  concerning  the  sequential  tests 
employed  in  the  field  maintenance  of  a specific 
symptom.  This  will  provide  a sample  of  actual  trouble- 
shooting sequences  which  could  be  compared  to  various 
projected  strategies.  In  this  manner,  one  would  be 
able  to  test  the  validity  of  several  predicted 
approaches.  What  does  the  maintenance  man  try  to 

make  optimum:  down  time,  nimiber  of  tests,  part 
procurement  trips,  his  evaluation  record? 

On  the  other  hand,  one  does  not  have  to  know  why  things 
are  not  optimum  in  order  to  Improve  the  situation.  Of  course 
the  Improvement  could  be  based  on  a sounder  rationale  if  the 
specific  areas  of  suboptimization  were  known,  but  a general 
sweeping  program  could  be  of  benefit  anyway.  The  Symptom 
Matrix  provides  a set  of  implementation  potentials  which  could 
be  constructed  for  some  small  piece  of  equipment,  about  the 


*Thls  document  described  a technique  for  predicting  symptom  and 
discussed  the  utilization  of  such  predictions.  All  the  predic- 
tions of  this  report  which  could  have  been  tested  by  the  present 
study  were  confirmed  by  the  present  study. 


complexity  of  a color  TV  set.  From  this  matrix,  design 
recommendations  for  packaging  and  monitoring,  training  aids 
and  trouble-shooting  aids  should  be  developed  and  compared  to 
current  practice.  This  direct  approach  to  the  problem  of 
maintenance  efficiency  Is  warranted  In  light  of  (a)  the  need 
for  Improved  maintenance  and  (b)  the  body  of  evidence  supporting 
a Symptom  Matrix  approach. 
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APPENDIX  I 


B-52  DATA:  WALKER  APB 


Variable 

Variable  Exj 

)lanatlon 

1. 

A/C  Identification 

Base  originated  code  number 

2. 

Flight  Number 

Consecutive  within 

each  A/C 

3. 

Date  of  Event 

Day,  Month,  Year 

4. 

Type  of  Event 

Assigned  Code: 

1.  Flight  and  subsequent  associated 
maintenance 

2.  Maintenance  to  correct  additional 
malfunctlon( s ) detected  during 
post -flight  maintenance 

3.  Maintenance  following  operator 
pre -flight  complaint 

4.  Maintenance  to  correct  mainte- 
nance observed  malfunctions 
other  than  those  covered  by- 
code  2 

5.  Test  Control  Report 
■*■8.  Supplementary  Report 

9.  Other 

5.  Type  of  Mission  Assigned  Code: 

1.  RBS,  Radar 

2.  Camera,  Attack 

3.  Navigation  Only 

5.  RBS,  Radar  and  Camera 

5.  RBS,  Radar  and  Other 

6.  Camera  Attack  and  Other 
**9.  Other 


Att  XT«  fao 


6.  Length  of  Mission  To  nearest  1/10  hour 

7*  System  Type  Assigned  Code: 

A ARC -65 
B ARC -34 
C AIC-10 
D ARA-25 
E APX-25 
F ARN-21 
G APN-21 
H MD-1 


J ASB-4 
K ASB-4A 
L ASB-9 


Z System  not  under 

surveillance 

8. 

System  Number 

For  systems  used  In 
the  A/C 

multiples  on 

9. 

System  Rating 

Flight  crew  rating. 

assigned  code: 

0.  Not  used 

1.  No  maintenance  required 

2.  Maintenance  required 

3.  Training  lost  due  to  this  system 

4.  Flight  cancelled  due  to  this 
system 

5.  Flight  cancelled  due  to  another 
system  under  surveillance 

6.  Flight  cancelled  due  to  system(s) 
not  under  surveillance 

*9.  Other 

10.  Symptom  4-dlgit  code  (See  Appendix  C) 

11.  Maintenance  Action  taken,  assigned  code: 

1.  Checked;  not  trouble  found 

2.  Repaired  by  unit  replacement 

3.  Repaired  by  Sub-unit  replacement 

4.  Repaired  by  Part  replacement 

5.  Repaired  by  part  adjustment 

6.  Repaired  by  part  repair 

7.  Modified,  cannibalized,  switched, 
etc. 

8.  Repair  action  performed  outside 
system 

9.  Other 


♦Numbers  (7)  and  (8)  not  used  as  codes  here. 
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A.  2-3  (le.,  both 

B.  2-4 

C.  2-5 

D.  2-6 

E.  3-4 

F.  3-5 

G.  3-6 

H.  4-5 

J.  4-6 

K.  5-6 

L.  3 or  more  code 
N.  2-8 

P.  3-8 

R.  4-8 

S.  5-8 

T.  6-8 


--  as  given  above) 


numbers 


12.  Unit  4-dlgit  code  (last  digits  of  manu- 

facture designation;. 

13.  Unit  Maintenance  Action  taken,  assigned  code: 

I.  Checked;  no  trouble  found 

3.  Repaired  by  subunit  replacement 

4.  Repaired  by  part  replacement 

5.  Repaired  by  part  adjustment 

6.  Repaired  by  part  repair 

J.  Switched,  modified,  cannibalized, 
etc. 

*9.  Other 

E.  3-4 

F.  3-5 

G.  3-6 

H.  4-5 

J.  4-6 

K.  5-6 

L.  3 or  more  code  numbers 


14. 

Serial  Out 

Last  four  digits 
serial  number 

of 

removed  unit 

15. 

Serial  In 

Last  four  digits 
number 

of 

new  unit  serial 

16. 

Sub-Unit 

4 -digit  code 

♦Number  (8)  not  used  as  a code  here. 
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17.  Sub-Unit  Maintenance  Action  taken,  assigned  code; 

I.  Checked;  no  trouble  found 

4.  Repaired  by  part  replacement 

5.  Repaired  by  part  adjustment 

6.  Repaired  by  part  repair 

7.  Switch,  modified,  cannibalized, 
etc. 

H.  4-5 

J.  4-6 

K.  5-6 

L.  4-5-6 


00 
i — 1 

Serial  Out 

Last  4 digits  of  removed  sub-unit 
serial  number 

19. 

Serial  In 

Last  4 digits  of  new  subunit  serial 
number 

• 

o 

OJ 

Part  Circuity  Symbol 

7 -digit  circuit  symbol 

21. 

Part  Name 

6 -digit  code 

22. 

Part  Defect 

3 -digit  code 

23. 

Part  Maintenance 

Action  taken,  assigned  code: 

*4.  Replaced 

5.  Adjusted 

6.  Repaired 

7.  Modified,  cannibalized,  etc. 
9.  Other 

K.  5-6 

M.  Complete  standard  alignment 


24. 

Shop/Depot  Date 

Day,  Month,  Year  of  completion  at 
this  level 

25. 

NRTS  (Not  Repairable 
This  Station; 

Reason,  assigned  code: 

1.  Unit  sent  to  depot  for  repair 

2.  Subunit  sent  to  depot  for  repair 

3.  Unit  repair  attempted;  unsuccesful; 
sent  to  depot  for  repair 

4.  Subunit  repair  attempted; 
unsuccessful;  sent  to  depot  for 
repair 


* Numbers  (l)  through  (3)  and  number  (8)  not  used  as  codes  here. 
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5.  Unit  scrapped 

6.  Subunit  scrapped 

7.  Same  as  2,  use  If  not  first 
subunit  reported 

8.  Same  as  4,  use  If  not  first 
subunit  reported 

26.  Report  Number  Consecutive  for  each  A/C 

27.  Malfunction  Number  Consecutive  within  a report 

28.  Reason  for  Maintenance  Numbers  for  flight  line;  letters 

for  shop,  assigned  code 

1 (or  k)  To  correct  operator  complaint 

2 (or  b)  To  correct  maintenance 

complaint 

3 (or  C^  Periodic  check 

4 (or  Dj  Modification,  recycle 

5 (or  E)  Activity  following  operator 

In-fllght  maintenance 

6 (or  P)  To  correct  pref light 

complaint 

7 (or  G)  To  verify  complaints  only 

8 (or  H)  Cannibalization  only 

9 (or  j)  Unknown 


29. 

Malfunction  Verified? 

Yes,  No, 

N/A 

30. 

Number  of  Men 

Number  of 

7 level  technicians 

31. 

Preparation 

Activity 

1/lOOths 

code; 

time 

In 

hours 

and 

32. 

Malfunction 

Verification 

Activity 

1/lOOths 

code; 

time 

In 

hours 

and 

33. 

Fault  Localization 

Activity 

1/lOOths 

code; 

time 

In 

hours 

and 

34. 

Part  Procurement 

Activity 

1/lOOths 

code ; 

time 

In 

hours 

and 

35. 

Repair 

Activity 

1/lOOths 

code; 

time 

In 

hours 

and 

36. 

Final  Malfunction  Test 
Time 

For  each 

malfunction,  clock  time 

37. 

Final  System  Test  Time 

For  each 

system,  clock 

time 
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38.  Time  System  Available  Clock  time  to  nearest  hour  that 

flight  ended 

39.  "On  time"  Filament  or  standby  time  accumulated 

during  maintenance 

40.  Administrative  Time  Kind  (coded  as  given  below),  time 

In  hours 

1.  System  Administrative  time 

2.  Malfunction  administrative  time 

3.  System  and  malfunction  logistic 
time 

4.  Unknown 

41.  Logistic  Time  Code  for  kind  of  time,  and  time  In 

hours  and  l/lOOths  of  hours 

1.  System  logistic  time 

2.  Malfunction  logistic  time 

3.  System  and  malfunction  logistic 
time 

4 . Unknown 

42.  Initial  Delay  Time  Hours  between  time  system  became 

available  and  time  active  repair 
started 

In  addition,  verbatim  symptoms  were  elaborately  (linguistically) 
coded  - see  Appendix  D 
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APPENDIX  II 


1. 


2. 


3. 


4. 


5. 


6. 


7. 


8. 


9. 


10. 


11. 


12. 


F-106  DATA;  DOVER  AND  SELFRIDGE  AIR  FORCE  BASES 


Variable 

Base 

Squadron 

Pilot  Number 

Aircraft  Number 
Aircraft  Report  Number 


Variable  Explanation 

Dover  vs.  Selfrldge 

95th  Fighter-Interceptor  Squadron 
(FIS)  - Dover; 94th  and  71st  (FIS)  - 
Selfrldge 

Locally  assigned;  not  necessarily 
unique 

Last  3 digits  of  serial  number 

Sequentially  assigned  by  event 
within  A/C  number 


Date 


Day,  Month,  Year 


Cumulative  Flight  Hours  On  this  A/C 
Mission  Type  Assigned  code: 

1.  Active  ADC  missions 

2.  Intercept  sorties  during  tactical 
evaluations  and  Operational 
Readiness  Inspections  (ORI's). 

3.  Intercept  sorties  during  other 
exercises 

4.  Intercept  sorties  during  normal 
training 

5.  Functional  check  flights 

6.  All  other  missions 

Mission  Success  Pilot's  Rating  of  System  Readiness  - 

1»  2, or  3 


Mission  Evaluation 


Aborted  (where)  or  other 


Passes  Attempted  Number  of  Intercepts  attempted 

during  mission 

Passes  Completed  Attempted  minus  number  of  symptoms 

reported 


Symptoms  classified  Into  7 general 
areas 


13.  Report  Type 


14.  Reason  for  Report 

Assigned  code: 

1.  Flight 

2.  Air  Abort  - flight  control  system 

3.  Air  Abort  - MA-1  system 

4.  Air  Abort  - Not  FSC  or  MA-1 

5.  Ground  Abort  - FCS 

6.  Ground  Abort  - MA-1 

7.  Ground  Abort  - Not  FSC  or  MA-1 

8 Maintenance  Originated  Complaint, 
Unscheduled 

9 Maintenance  Originated  Complaint 
during  Scheduled  Maintenance 

15-  Malfunction  Number 

Consecutively  within  A/C  Report 
Number 

16.  Symptom  Code 

From  76-3  Form  as  reported  by 

ADCR- 

17.  When  Discovered 

Taken  from  AFM  66-1  codes 

18.  Severity  of  Complaint 

ARINC  Research  Field  Engineer's 
rating  - 1,  2,  3 or  4 

19*  Verification  of 

Assigned  codes: 

Complaint 

1.  Specific  Operator  Complaint 
Verified 

2.  Not  Verified,  other  trouble 
found  In  some  subsystem 

3.  Not  Verified,  other  trouble  In 
other  subsystem 

4.  Maintenance  Originated  Complaint 

5.  Not  checked 

6.  No  data  available 
*9  No  trouble  found 

20.  Action  Taken  From  T.O.  IF-IO6-O6  code 

21.  Maintenance  Concept  Assigned  code: 

1.  Team  Concept  Used 

2.  B’C&M  Personnel  only 

3.  MA-1  Personnel  only 

4.  Mechanical  Personnel  only 

5.  Any  of  the  above  plus  Technical 
Rep.  Assistance 


* Numbers  (7)  and  (8)  not  used  as  codes  here. 
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1 


Assigned  code: 


22.  Method  of  Trouble 
Shooting 

1.  Self  test  features 

2.  Unit  substitution 

3.  Wholesale  unit  substitution 

4.  Special  test  equipment 

5.  Short  system  ground  check 

6.  Multimeter  testing 

7.  Locally  built  test  devices 

8.  Visual  Inspection 

9.  Other 

A Steering  loop  alignment 
B Mark-up  check 

23.  Next  Flight  Success  Assigned  code: 

0.  Unknown  or  indeterminate 

1.  Successful 

2.  Not  successful  - similar  symptom 

3.  Not  successful  - new  symptom 

4.  Failed  prior  to  next  flight 

5.  Not  applicable 

24.  AFCS  Subsystem  Downtime  Total  maintenance  delay  time  In 

tenths  of  an  hour 

25.  Work  Unit  Code  From  T.O.  IF-IO6-O6  code  for  each 

unit  Involved 

26.  Flight  Line  Action  for  Assigned  code: 

Unit 

0.  TCTO  or  special  test  requirement 

1.  Adjust  unit  In  place 

2.  Remove  - replace,  no  adjustment 
required 

3.  Remove  - replace,  adjustment 
required 

4.  Remove  - reinstall,  no  trouble 
found  In  shop  check 

5.  Remove  - reinstall,  repaired  In 
shop 

6.  4 plus  additional  adjustment  In 
A/C 

7.  5 plus  additional  adjustment  In 

a/c 

8.  Replace  parts  or  assemblies  In 
place 

9.  Make  other  repairs  In  A/C 

27.  Unit  Serial  Number  Last  4 digits  of  serial  number 


28.  Unit  Report  Number  Consecutively  within  serial  number 

within  W.O.C. 

29.  Unit  Cumulative  Hours  Estimated  total  operating  hours 

30.  How  Malfunction  (Unit)  Symptom  to  shop  from  APTO  210/211 

form 

31.  Reason  for  Action  (Unit)  Assigned  code: 

0.  Termination 

1.  Suspected  unit  malfunction 

2.  Removed  for  bench  check 

3.  Removed  as  one  unit  of  a group 
swap  action 

4.  Removed  during  scheduled  mainte- 
nance 

5.  Removed  for  special  Inspection 
or  TCTO 

6.  Installed,  OK 

7.  Installed,  Not  OK 

32.  When  Discovered  (Unit)  From  T.O.  IF-IO6-O6  code  list 

33.  Action  Taken/Dlsposltlon  Assigned  code: 

(Unit)  Reseated  sub-unit 

1.  Adjust  only 

2.  Replace  subunits 

3.  Replace  parts  (electrical) 

4.  Replace  parts  (hardware) 

5.  Combinations  of  2,  3,  and  4 

6.  Repair  wiring,  connectors,  or 
solder  joints 

7.  Clean  and/or  lubricate 

8. 

9.  No  trouble  found 

A.  NRTS,  lack  of  parts 

B.  NRTS,  lack  of  technical  skill 

C.  NRTS,  lack  of  test  equipment 

D.  NRTS,  sealed  unit 

E.  NRTS,  beyond  local  repair 

Assigned  code: 

0.  MAAMA  testers 

1.  Test  stand  procedures 

2.  Special  test  equipment 

3.  Locally  built  test  devices 

4.  Ohmmeter,  resistance  testing 

5.  Visual  Inspection 

6.  Other 

7.  TRT  Tester 

8.  Pr-evlous  experience 

9.  96089-801  tester 


34.  Method  of  Fault 
Isolation  (Unit) 
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35»  Unit  Active  Repair  Time  Manhours,  for  shop.  In  tenths  of 

an  hour 

36.  Flight  Line  Personnel  1 Name  code  (Dover  only)  of  mainte- 

nance supervisor  for  this  action 

37.  Flight  Line  Personnel  2 Name  code  (Dover  only)  of  mainte- 

nance man  ^ on  this  action 

38.  Squadron  - Flight  A,  B,  or  C according  to  work  shift 
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APPENDIX  III 


SYMPTOM  REPORTING 


1.  INTRODUCTION 

This  appendix  describes  how  symptoms  are  currently  being 
reported  In  the  Air  Force  maintenance  system  and  also  how  they 
were  being  reported  during  the  data  collection  periods  for  the 
data  studied  here.  In  order  to  do  this,  the  appendix  Is  divided 
Into  four  parts.  The  first  part  deals  with  current  Air  Force 
regulations  concerning  symptom  reporting  as  contained  In  AFM  66-1. 
The  second  part  covers  symptom  reporting  procedures  at  Walker  AFB 
(SAC)  from  March  22,  i960  until  April  6,  1961  — the  period  during 
which  the  primary  ASB-4  data  considered  In  this  study  was  collected. 
The  third  part  examines  symptom  reporting  procedures  at  Dover  APB 
(ADC)  from  November  1965  to  February  1966  — the  period  during 
which  data  concerning  P-IO6  electronic  subsystems,  also  Included 
In  the  study,  was  collected.  The  fourth  part  contains  brief 
description  of  current  reporting  procedures  at  certain  other  Air 
Force  Bases  from  which  first  hand  experience  and  Information  are 
available. 

2.  MAINTENANCE  PROCEDURES  RELEVANT  TO  SYMPTOM  REPORTING  SPECIFIED 
IN  AFM  66-1  (1965 

Aside  from  prescribing  a very  general  organizational  struc- 
ture within  which  symptoms  are  reported,  AFM  66-1  does  not  specify 
the  specific  communication  nets  to  be  employed  by  Individual  bases. 
The  one  procedure  relevant  to  symptom  reporting  which  AFM  66-1 
does  specify  Is  the  use  of  post  flight  debriefing.  The  regulation 
specified  that; 

"For  those  major  commands  operating  complex 
weapons  systems,  an  operator-maintenance 
debriefing  procedure  will  be  utilized. 

Through  detailed  debriefings  of  operating 
crews  experience  and  complete  analysis  of 
reported  discrepancies,  many  maintenance 
manhours  may  be  saved  by  knowing  what  the 
trouble  Is  and  what  Is  required  to  correct 
it  prior  to  scheduling  maintenance  action. " 

(Section  2 paragraph  23) 


^ AFM  66-1  was  revised  15  July  1966,  after  this  study  was  underway; 
reference  Is  being  made  here  to  the  preceding  edition.  The 
procedures  have  not  been  materially  changed  In  the  newer  edition. 
Distribution  copies  were  not  available  when  this  draft  was 
prepared. 


This  clearly  places  the  operator-maintenance  debriefing  in  the 
central  position  as  far  as  routine  symptom  reporting  is  concerned, 
(It  is  also  obvious  from  the  above  paragraph  that  symptoms  are 
considered  an  Important  factor  in  maintenance  efficiency. ) 

AFM  66-1  does  indicate  that  the  reported  symptoms  (from  the 
debriefing  session)  should  go  directly  to  maintenance  control, 
and  from  there  to  the  flight  line  maintenance  personnel  (via  the 
maintenance  supervisor). 

In  general  then,  the  prescribed  chain  of  symptom  Information 
is:  ope rat or -maintenance  debrlefer  - maintenance  control  - 
maintenance  supervisor  - flight  line  maintenance  personnel.  Other 
information  channels  are  not  precluded,  but  they  are  left  up  to 
the  discretion  of  individual  commands  and/or  bases.  There  is  a 
certain  amount  of  leeway  in  the  details  of  the  symptom  reporting 
situation. 

ADC  Sup.  1 to  AFM  66-1  provides  more  detailed  recommendations 
concerning  the  specific  ways  in  which  AFM  66-1  regulations  are 
to  be  Implemented  for  ADC  commands: 

"The  form  specifies  that  a complete  debriefing 
team  must  be  available  for  all  missions,  and 
that  a complete  debriefing  team  consists  of  at 
least  the  following  personnel : 

(1)  An  aircrew  member  highly  skilled  and 
experienced  in  the  aircraft  and  qualified 
to  Interpret  malfunctions  in  the  weapons 
systems. 

(2)  An  APG  technician 

(3)  An  A & E (Armament  and  Electronics) 
technician 

When  applicable  or  required,  a radar  Intercept  officer  and 
an  engine  technician  are  added  to  the  debriefing  team.  ADC  Sup.  1 
makes  special  note  that  minimum  requirements  for  F-106  units 
"will  Include  weapons  control  and  integrated  system  technicians. " 

In  addition  to  the  above,  crew  chiefs  are  encouraged  to  participate 
in  the  debriefing.  The  senior  maintenance  man  on  the  team  is  the 
debriefing  team  supervisor,  and  it  is  his  responsibility  to  see 
that  the  debriefing  is  properly  conducted.  The  members  of  the 
debriefing  team  usually  return  to  their  normal  maintenance  duties 
when  they  are  not  required  for  debriefing  duties.  ADC  Sup.  1 
specifically  states  that  "all  members  of  the  debriefing  team  will 
be  fully  qualified  in  their  specialties". 

A cursory  debriefing  is  to  be  conducted  during  approach, 
landing,  or  shortly  after  landing  by  UHP  radio  or  flight  line 
Intercom.  The  minimum  information  to  be  conveyed  at  this  time 
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must  include  relaying  the  status  code^  of  any  malfunctioning 
system  to  workload  control.  Immediately  following  the  termination 
of  any  flight  (or  attempt)  pilots  and  crew  go  directly  to  the 
debriefing  area,  taking  aircraft  forms  with  them.  During  directed 
exercises  a modification  of  routine  debriefing  procedure  is 
sometimes  instituted.  After  a step-by-step  debriefing,  the 
debriefing  documents  are  sent  to  workload  control  as  soon  as 
possible. 

The  regulations  specify  that  "The  procedures  used  in  debrief- 
ing must  be  continuously  reviewed  to  Insure  that  aircraft  are 
turned  around  with  minimum  delay",  and  that  all  documents  reflect- 
ing system  verification  status  be  "reviewed  and  reconciled"  monthly. 

Even  detailed  supplements  to  AFM  66-1,  such  as  the  above, 
do  not  have  provisions  for  non-routine  methods  of  symptom  reporting. 
The  basic  information  channel  is  still  operator  - debrlefer  - 
maintenance  control  - maintenance  supervisor  - flight  line  mainte- 
nance personnel. 

3.  SYMPTOM  REPORTING  AT  WALKER  APB  (SAC)-ASB-4  SYSTEM 

The  bombing  navigator  records  in  his  flight  log  (Form  #781) 
whether  the  system  operated  satisfactorily  or  malfunctioned  during 
the  mission.  If  he  reports  that  the  system  did  not  operate 
satisfactorily,  he  also  records  the  symptom(s)  of  the  malfunction 
and  whether  or  not  in  flight  maintenance  was  attempted  and 
successful. 

After  landing  the  crew  is  debriefed  Immediately  by  skilled 
representatives  of  the  various  maintenance  shops.  The  A & E 
debrlefer  reviews  the  written  flight  log  and  also  questions  the 
bombing  navigator  in  an  effort  to  get  precise  Information  concern- 
ing any  malfunctions  which  occurred.  The  debrlefer  then  combines 
the  data  recorded  in  the  flight  log  with  the  additional  verbal 
Information  obtained  during  this  Interogatlon  of  the  navigator. 

This  combined  description  of  malfunction  symptoms  Is  recorded  on 
the  Job  order  form  992  which  Is  sent  to  work  load  scheduling 
and  the  maintenance  analyst. 

Work  load  scheduling  uses  the  Information  on  the  Job  order 
form  to  schedule  work  time  so  that  congestion  Is  avoided,  and 
sufficient  electric  and  hydraulic  power  Is  available  for  all 
maintenance  actions.  After  the  schedule  for  the  maintenance 


^There  are  three  codes:  1.)  aircraft  ready  for  next  mission  with 
servicing,  2.)  aircraft  capable  of  flying  a mission,  but  some 
equipment  modes  are  non-operable,  and  3.T  aircraft  cannot  fly 
another  mission  until  maintenance  Is  performed. 
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action  is  determined,  this  information  (the  schedule  for  the 
maintenance  action)  - and  a copy  of  the  Job  order  fonn  - are 
sent  directly  to  the  maintenance  personnel  who  will  ultimately 
troubleshoot  the  symptom(s).  The  maintenance  personnel  accomplish 
the  work  necessary  to  correct  the  symptoms  associated  with  the 
malfunctions  listed  on  the  job  order  form,  and  they  are  respon- 
sible for  rechecklng  any  In-fllght  maintenance  which  may  have 
been  attempted.  The  verification  status  of  symptoms,  and 
Information  about  what  maintenance  actions  were  taken,  are  sent 
to  the  maintenance  analyst. 

The  maintenance  analyst  keeps  records  of  all  symptoms 
reported  and  all  work  done,  and  he  keeps  a performance  history 
of  each  A/C.  The  maintenance  analyst  reports  general  maintenance 
trends  and  problems  to  the  maintenance  supervisor  and  to  the 
work  load  scheduling.  The  maintenance  supervisor.  In  turn, 
conducts  dally  "stand  up"  meetings  with  the  maintenance  personnel 
to  review  current  maintenance  problems  and  to  assess  the  degree 
of  maintenance  effectiveness. 

There  are  times  when  the  above  "routine"  channels  of  Informa- 
tion are  not  sufficient  for  effective  maintenance.  This  Is 
especially  true  when  unusual  maintenance  problems  occur,  such 
as:  repeat  non-verlfled  complaints,  no  trouble  found.  Intermit- 
tent and  self-corrected  malfunctions,  etc.  When  this  happens, 
the  maintenance  personnel,  maintenance  supervisor  and/or  mainte- 
nance analyst  may  communicate  directly  with  the  system  user 
(Bombing  navigator)  or  debrief er.  It  should  be  noted  that  such 
direct  communications  are  not  routine  and,  therefore,  are  not 
recorded  In  the  data  used  In  this  study. 

In  summary,  symptom  Information  comes  to  the  flight  line 
malntenai.ee  technician  from  some  or  all  of  the  following  sources: 

(1)  Information  recorded  during  debriefing  on  Job  order 
form. 

(2)  Meetings  with  maintenance  supervisor,  which  may  use 

(a)  Information  from  workload  control 

(b)  Information  from  maintenance  analyst 

(c)  Supervisor  experience 

(3)  Direct  communication  with  system  user  or  debrlefer. 

(4)  Symptoms  observed  during  troubleshooting  or  malfunction 
verification. 


73 


4.  SYMPTOM  REPORTING  AT  DOVER  AIR  FORCE  BASE  (ADC): 

ELECTRONICS  SUBSYSTEMS 

As  the  aircraft  Is  preparing  to  land,  or  shortly  after 
landing  and  during  taxi,  the  pilot  Is  In  radio  contact  with 
Maintenance  Control. 

The  pilot  gives  a verbal  description  of  all  symptoms  to  the 
chief  of  Maintenance  Control.  The  chief,  who  Is  assisted  by  a 
number  of  additional  personnel.  Is  usually  a highly  skilled 
ex -mechanical  technician.  Maintenance  Control  decides,  tentatively. 
If  quick  turn  around  of  the  aircraft  Is  possible.  If  cursory 
Inspection  might  be  fruitful  (usually  a matter  of  course),  and 
what  general  work  facilities  will  be  required.  These  decisions 
are  based  on  experience,  as  there  are  no  fixed  routines  for  each 
set  of  possible  symptoms. 

Under  most  conditions  the  aircraft  taxis  to  the  cursory 
Inspection  area,  where  flight  line  technicians  have  been  waiting 
and  where  they  can  talk  to  the  pilot  via  Intercom.  (Prior  to 
hooking  up  the  Intercom,  the  flight  line  technician  had  been  In 
Indirect  contact  with  the  pilot  via  relayed  messages  from  Mainte- 
nance Control. ) Flight  line  personnel  remove  the  Radar  film 
pack  and  send  It  back  to  the  photo  shop  for  Immediate  processing. 

The  flight  line  electronics  technician,  usually  a senior  skilled 
technician,  may  then  examine  the  aircraft's  electronic  subsystems 
while  they  are  still  being  supplied  with  aircraft -generated 
power.  He  has  15  minutes  to  diagnose  and  fix  the  malfunction  If 
he  can.  If  he  can  not  fix  It,  or  If  he  can't  diagnose  It,  or  If 
he  can  not  even  verify  the  existence  of  the  malfunction,  then  he 
decides  what  facilities  are  required  and  assists  Maintenance 
Control  In  selecting  a location  for  parking  the  aircraft.  If 
the  malfunction  was  repaired  In  cursory,  the  aircraft  can  be 
turned  around  quickly,  and  what  follows  In  time  Is  merely  paper 
work.  If  the  aircraft  was  not  repaired  In  cursory,  then  there 
Is  always  further  work.  (The  technician  Is  not  allowed  to  "write- 
off" the  malfunction  without  action. ) 

After  cursory,  the  aircraft  Is  parked.  Maintenance  may 
begin  without  paperwork  If  the  maintenance  supervisor  for  that 
flight  thinks  It  Is  desirable  to  start.  Sometimes  he  was  the 
flight  line  technician  at  cursory  and  may  feel  that  he  has 
special  knowledge. 

When  the  aircraft  has  seen  parked,  the  pilot  goes  Immediately 
to  the  debriefing  area  where  he  and  other  pilots  from  that  flight 
are  met  by  operations  and  maintenance  personnel  and  perhaps  by  a 
Hughes  technical  representative.  During  debriefing,  the  film 
record  of  the  radar  scope  action  Is  run,  and  an  'M.-1  (fire  control 
system)  flight  line  maintenance  officer  (skill  level  E7  or  above) 
questions  the  pilot  about  his  electronics  subsystems  complaints. 
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A 76-3  debriefing  form  and  the  78I  aircraft  record  are  filled 
out  at  this  time.  An  effort  Is  made  to  define  the  pilot's 
complaint  In  terms  of  a 3 -letter  symptom  code  from  the  ADCR  66-28 
code  book.  The  maintenance  officer  and  the  pilot  try  to  agree 
on  a symptom  designation,  which  Is  sometimes  arbitrated  by  the 
maintenance  officer  - especially  If  the  pilot  Is  Inexperienced, 
or  unfamiliar  with  the  equipment.  Some  supplementary  narrative 
description  may  accompany  the  3-letter  code,  especially  In  the 
case  of  ambiguous  or  unusual  descriptions. 

A carbon  copy  of  the  78I  form  and  one  copy  of  the  76-3  form 
are  sent  to  maintenance  control,  where  Job  specialist  requirements 
are  allocated.  Another  copy  of  the  76-3  Is  taken  to  the  flight 
line  officer's  dispatch  area  by  the  debrlefer,  and  given  to  the 
flight  supervisor.  The  latter  copy  Is  really  for  the  supervisor's 
Information  only,  since  the  actual  maintenance  authorization  comes 
from  Maintenance  Control.  Maintenance  Control  telephones  the 
flight  supervisor  at  the  flight  line  officer's  dispatch  area  and 
Itemizes  the  Job  specialist  requirements  for  all  MA-1  maintenance 
actions.  (Maintenance  Control  completes  a form  992).  The  flight 
line  supervisor  allocates  all  this  work  among  his  personnel. 

The  man  assigned  to  a malfunction  may  (40^)  or  may  not  (60^)  have 
been  at  the  cursory  Inspection.  In  any  case  he  also  receives  a 
copy  of  the  76-3  form  and  may  also  receive  a supervisor's 
recommendation:  "replace  unit  so  and  so  from  truck  stock." 

During  his  troubleshooting  operations,  the  maintenance  man 
records  all  of  the  symptoms  which  ^ observes  on  an  AFTO  210 
form,  and  also  records  each  unit  he  removes  (along  with  reasons 
for  removal)  on  an  AFTO  211  form.  These  reasons  are  subsequently 
coded  as  "How  Mai"  codes  and  are  the  symptoms  received  by  the 
shop  technician  when  he  troubleshoots  the  removed  units. 

In  summary,  the  Information  available  to  the  man  who  Is 
involved  In  the  actual  maintenance  action  can  come  from  some  or 
all  of  the  following  sources: 

(1)  Second  hand  report  of  the  radio  communication  between 
the  pilot  and  Maintenance  Control. 

(2)  Direct  Intercom  or  face-to-face  contact  between  pilot 
and  flight  line  technicians  during  cursory  Inspection. 

(3)  Symptoms  observed  during  cursory  Inspection. 

(A)  Information  recorded  during  debriefing. 

(5)  Supervisor's  recommendations. 

(6)  Symptoms  observed  during  maintenance  action. 
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5.  CURRENT  SYMPTOM  REPORTING  AT  OTHER  BASES 

5.1  SelfrldKe  AFB  (F-IO6) 

The  procedures  here  are  much  like  those  used  at  Dover, 
except  as  follows.  Upon  landing,  the  pilot  goes  to  debriefing 
where  he  Is  debriefed  separately  for  MA-1  and  AFC.  He  narrates 
his  complaint  to  the  debrlefer  who  enters  three  letter  codes 
and  amplifying  remarks  (if  necessary).  The  sheet  Is  then  sent 
to  maintenance  control,  then  to  the  shop,  and  then  the  repairman. 
If  an  exercise  Is  In  progress  the  pilot  may  radio  the  symptom 
to  maintenance  control  during  taxi -In.  Control  then  radios  a 
spares  truck  which  meets  the  aircraft  as  It  parks,  and  attempts 
repairs.  The  pilot  must  still  go  through  debriefing. 

5.2  MacDlll  AFB  (F-4C) 

After  each  flight  the  aircrew  Is  Interviewed  by  a representa- 
tive of  each  shop  to  discuss  the  various  system  failures  In 
detail.  The  debrief ers  are  usually  airmen  first  class  or  a staff 
sargeant  of  either  five  or  seven  level  qualifications.  The 
debrlefers  first  discuss  the  symptoms  observed  by  the  aircrew  to 
determine  that  operator  error  didn't  cause  the  observed  malfunc- 
tion. If  It  Is  determined  that  operator  error  was  not  the  cause, 
then  specific  questions  are  asked  to  assist  In  locating  the 
trouble  more  readily. 

When  the  aircrew  has  been  debriefed,  the  complaints  are 
transcribed  on  a special  machine  and  transmitted  to  maintenance 
control.  When  maintenance  control  receives  the  complaints  It 
assigns  a work  order  and  a time  to  start  the  repair  action  to 
the  appropriate  shop.  Then  Maintenance  Control  verbally  calls 
the  failure  Information  to  the  respective  shops  to  effect  repair 
of  the  failed  system.  Rarely  will  the  maintenance  men  that 
actually  repair  a system  discuss  a problem  with  the  aircrew  that 
makes  the  complaint.  If  a discussion  Is  held  between  maintenance 
men  and  aircrew  It  will  only  be  after  repeat  writeups  on  the 
same  aircraft,  same  system  and  same  complaint.  Then  It  will 
probably  be  a supervisor  rather  than  the  technicians  who  will 
effect  the  repair. 

5 . 3 Little  Rock  AFB  (B-58) 

People  from  the  shops  meet  the  aircraft  and  get  a quick 
word  from  the  crew;  the  latter  then  report  Immediately  to 
debriefing  where  a code  and.  If  necessary,  brief  narrative  are 
taken  down  by  a senior  NCO  (who  has  the  code  book  memorized). 

One  copy  of  the  form  (126-1)  goes  to  the  shop  and  one  to  Job 
control.  The  shop  gives  the  Job  to  a repairman  who  goes  out 
after  Job  control  notifies  the  shop  that  It  Is  "their  turn"  to 
effect  repairs. 
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APPENDIX  IV 


SYMPTOM  CODING 


1.  INTRODUCTION 

This  appendix  describes  how  both  ASB/4  flight  line  and 
F-106  shop  symptoms  were  coded  for  machine  analysis  In  this 
study.*  The  coding  procedure  was  derived  for  this  analysis 
and  as  such  may  be  thought  of  as  still  In  the  feasibility  stages. 

The  problem  In  coding  symptoms  Is  one  of  developing  a 
numerical  code  which  will  accept  all  of  the  vagaries  and  Idio- 
syncrasies of  unstructured  verbatim  material  and  yet  not 
destroy  those  characteristics  of  the  symptom  which  convey  the 
meaning.  Of  course  one  does  not  know  what  those  characteristics 
are  until  he  completes  the  analysis  for  which  the  code  Is  needed. 
As  a result,  all  uniqueness  has  to  be  coded  In  such  a way  that 
similar  numerical  codes  Imply  similar  symptom  patterns,  but  so 
that  Individuality  Is  also  maintained.  (it  was  decided  that 
there  should  be  no  compromise  with  the  goal  of  retaining  every 
distinction  available  In  the  data;  It  Is  only  by  retaining 
every  distinction  that  one  can  be  sure  he  has  retained  the 
Important  distinctions. ) 

2.  asb/4  symptoms 

Some  Idea  of  the  complexity  of  this  task  can  be  seen  In  the 
following  list  (Table  15)  of  symptoms  reported  on  one  ASB/4 
system.  (The  list  Is  relatively  representative  of  ASB/4  symptoms 
In  general. ) 

Many  symptoms  (that  Is  many  reports)  seem  to  refer  to  the 
same  general  problem,  but  each  symptom  (that  Is  each  specific 
complaint)  Is  unique. 

Each  symptom  was  separated  Into  Its  primary  grammatical 
constituents,  defined  as  (l)  Subject  or  subject  phrase,  (2) 
Description  or  verb  phrase,  (3)  Subject  modifiers,  and  (4) 
Qualifying  modlfers  phrases  and  Restrictive  modifier  phrases. 

This  separation  was  not  rigorous,  but  Is  was  consistent  within 
the  data  and  It  seems  to  have  been  adequate.  (Future  work 
should  be  well  advised  to  use  this  basic  attack  with  a more 
flexible  set  of  grammatical  functions. ) Table  l6  Illustrates 
what  this  decomposition  of  symptoms  actually  led  to. 


*For  the  F-106  flight  line  data  the  symptoms  which  were  coded 
during  data  acquisition  and  used  In  this  data  analysis  were 
the  3-letter  symptom  codes  from  the  76-3  forms.  (See  Appendix 
II).  Since  these  codes  are  being  revised  by  the  cognizant 
Air  Defense  Command  Office,  they  are  not  Included  here. 


TABLE  15 

LIST  OF  ASB/4  SYMPTOMS 
(26  Consecutive  Symptoms  From  System  N0004) 


1. 

VRM  wavy  vertex  jumps  In  off-center  2.  H-Cal 

10,000  ft  low 

2. 

Bomb  Bay  ring  out  due 

3. 

Stab  leveling  plate  cracked 

4. 

No  nav.  lltes  on  forward  panel 

5. 

Radar  Indicators  have  excessive  Interference, 
returns 

spoking,  weak 

6. 

Actvator  arm  warn 

7. 

No  Sweep 

8. 

Screws  missing  on  map  match  cover 

9. 

No.  15  second  delay  In  Memory  Point 

10. 

3000  ranging  error 

11. 

No  Sweep  Both  Scopes 

12. 

HA  will  not  settle  In  H-Cal 

13. 

C/I  ant.  Modulator  & RT  Unit 

14. 

P.  P.  Long  drives  Intermittently,  operator  states  that 
relay  60309  sticks 

15. 

Heading  & Az.  Mark  too  bright  on  10"  scope 

16. 

Broken  Wires  on  Cables  leading  to  Longitude  Computer 

17. 

Doubled  VRM  Close  range  In  Bomb 

18. 

HA  4500  feet  In  error 

19. 

No  Station  keeping  function  In  radar 

20. 

Radar  goes  to  recycle  then  back  to  radiate  In 
Slant  range 

H-Cal  and 

21. 

No.  3 Bracket  missing  on  radome 

22. 

Complete  post  dock  Inspection 

23. 

Cross  hairs  won't  track 

24. 

10"  Scope  loses  sweep  Intermlttantly 

25. 

Double  VRM 

26. 

Rlngout 
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TABLE  16 


In  coding  these  symptoms,  each  subject-verb  phrase  of  the 
symptom  (that  Is,  each  line  of  type  in  Table  l6)  was  coded  on 
one  IBM  card.  Preassigned  fields  on  the  card  were  set  aside  for 
coded  equivalents  for  (l)  the  subject  modifiers,  (2)  the  subject, 
(3)  the  description,  (4)  up  to  four  modifying  phrases,  (5)  the 
number  of  subject-verb  phrases  (that  is  the  number  of  cards) 
required  to  code  the  symptom*,  and  (6)  whether  or  not  the  basic 
word  pattern  of  the  symptom  as  it  is  coded  on  the  IBM  card  was 
forward  or  Inverted. 

After  analyzing  each  symptom  in  this  manner,  the  grammatical 
components  were  each  assigned  a four  digit  code.  The  codes  were 
assigned  in  such  a way  that  all  grammatically  (indeed  many 
typographically)  distinct  word  forms  were  coded  as  numerically 
distinct,  but  degrees  of  grammatical  similarity  were  represented 
by  similar  degrees  of  numerical  similarity.  For  example,  in  the 
four-digit  code  assigned  to  the  "subject"  position  on  the  IBM 
card,  the  first  two  digits  identified  the  ASB/4  subsystem  referred 
to.  The  third  digit  distinguished  between  separate  aspects  of 
that  subsystem  (if  it  has  separate  aspects);  the  fourth  digit 
varied  with  ways  of  referring  to  this  subject  and  thus  identified 
the  specific  word  or  words  employed  in  the  symptom.  Some  actual 
examples  of  the  subject  codes  employed  can  be  seen  in  Table  17. 


TABLE  17 


SAMPLE  SUBJECT  CODES 


Code 

Subject 

1800 

Warning  Lights  (Lltes) 

1820 

Coolant  Oil  Lite  (Light) 

1821 

Coolant  Oil  Lite  (Light) 

1822 

Coolant  Overhead  Lite  (Light) 

1823 

Coolant  Lite  (Light) 

1824 

Coolant  Overheat 

1841 

ECO  Lite  (Light) 

1842 

ECO 

1866 

Low  Limit  Lite  (Light) 

1867 

Lower  Limit  Lite  (Light) 

* Unfortunately,  the  code  lost  the  order  in  which  these  phrases 
occurred  in  the  symptom.  This  oversight  should  be  corrected 
in  future  language  analysis  study.  Sequential  dependencies 
are  undoubtedly  significant. 
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A careful  examination  of  these  examples  reveals  the  advantages 
of  this  method  of  coding.  First,  by  selecting  the  number  of 
fields  (l.e.  columns  on  an'  1W  ‘card)  on  which  to  sort,  subjects 
can  be  grouped  together  with  any  desired  degree  of  similarity. 

If  two  digit  subject  codes  are  used  then  all  subjects  referring  to 
"warning  lights"  will  be  grouped-  together  and  separated  from  sub- 
jects referring  to  all  other  ASB-4  subsystems.  By  adding  the  third 
digit  to  the  first  two,  the  discrimination  Is  extended  to  different 
kinds  of  warning  lltes;  if  the  fourth  digit  Is  utilized,  the  precise 
language  used  can  be  recovered.  Other  subjects  are  coded  In  an 
equivalent  manner.  The  second  advantage  In  the  coding  system  is 
that  the  symptom  retains  all*  of  Its  originality — the  information 
In  the  reported  symptom  Is  retained  Intact 

A similar  coding  process  was  used  for  the  other  grammatical 
elements  of  the  reported  symptom,  again  there,  the  first  one  or 
two  or  three  digits  referred  to  the  "meaning"  of  the  words  In  more 
and  more  detail  and  the  third  and  fourth  digits  referred  to  the 
specific  detail  or  form  used  In  that  symptom  report. 

3.  P-106  SYMPTOMS 

F-106  Shop  Symptoms  were  coded  In  manner  analogous  to  the 
one  used  for  the  ASB/4  flight  line  data,  but  the  codes  were 
different,  and  the  number  of  grammatical  components  considered 
was  larger.  Unfortunately,  time  was  not  available  for  a thorough 
analysis  of  these  data.  It  would  be  Interesting  to  know  If  this 
change  In  the  number  of  grammatical  components  represents  an 
Improvement  In  the  code  In  terms  of  Its  ability  to  parallel  the 
distribution  of  Information  within  symptoms. 


♦Except  for  the  failure  to  code  the  sequence  of  phrases  In  the 
symptom,  which  deficiency  was  corrected  when  the  F-106  shop 
symptoms  were  coded. 
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APPENDIX  V 


PROGRAMS  USED  IN  THE  ANALYSIS 

INTRODUCTION 

This  appendix  contains  the  source  programs  used  in  the 
analysis  of  data  for  this  study.  These  programs  were  used  on 
an  IBM  l401  computer  with  I6OOO  units  of  storage  and  tape  drives. 
These  programs  are: 


1. 

Univariate 

Information  Analysis 

2. 

Analysis 

of 

Variance 

3. 

Phase  I 

of 

Bivariate 

Information 

Analysis 

4. 

Phase  II 

of 

Bivariate 

Information 

Analysis 

Each  program  is  prefaced  with  an  instruction  sheet  Indicating 
how  the  program  is  to  be  used,  after  it  has  been  assembled.  This 
sort  of  form  facilitates  application  of  a program  with  which  the 

user  is  not  familiar or  a program  which  has  become  unfamiliar 

with  lack  of  use.  A flow  chart  of  the  univariate  analysis  is 
provided  for  those  desiring  to  recode  the  routine.  Since  general 
analysis  of  variance  routines  are  readily  available  for  larger 
machines,  there  was  little  point  in  flow  charting  the  routine 
used  here.  Supplementary  comments  on  the  input  and  output  for 
the  analysis  of  variance  are  Included.  The  following  paragraphs 
describe  how  the  input  data  are  sorted  before  making  a Phase  II. 

A description  of  the  Phase  II  output  is  also  given.  No  flow  chart 
is  provided  since  anyone  wanting  to  duplicate  the  bivariate 
analysis  on  a larger  machine  would  profit  from  completely  re- 
constructing the  computational  process.  Consequently,  the  con- 
ventional hand -calculation  paradigm  has  been  Included  to  indicate 
the  nature  of  the  calculations  being  performed  in  the  Phase  I and 
II  programs.  For  more  detailed  explanation  see  either  Attneave 
(ref.  5)  or  Garner  (ref.  3). 

Sorting  the  Input  Data  for  Bivariate  Infomnatlon  Analysis 

The  data  used  in  the  following  example  are  hypothetical,  but 
they  are  an  adquate  representation  for  Illustrating  the  sorting 
procedures  used  to  prepare  input  data  for  the  Phase  I routine.  To 
begin  with,  it  is  assumed  the  data  have  been  punched  on  IBM  cards 
in  several  fields  (groups  of  card  columns  used  to  code  a partic- 
ular element  of  data).  The  bivariate  analysis  routines  are 
concerned  with  the  entries  for  two  such  data  elements  or  fields 
(Table  I8). 
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RAW  DATA  ! 


Pilot  No. 

Symptom 

6l 

RHA 

72 

PAX 

91 

PA  A 

74 

RHH 

91 

PAY 

76 

PGA 

69 

RHA 

76 

RHH 

69 

PAX 

74 

RHH 

72 

FAX 

69 

FAA 

74 

RHA 

62 

PAX 

72 

RHH 

76 

PAY 

91 

RHH 

91 

RHA 

74 

RHA 

62 

PAX 

74 

PAY 

69 

RHA 

91 

RHH 

Consider  the  condition  where  one  wants  to  know  the  extent  to 
which  pilot  uncertainty  and  ssmiptom  uncertainty  are  related  or 
overlap  (as  per  Figure  2 in  the  text).  The  first  step  in  pre- 
paring the  input  to  the  Phase  I routine  is  to  order  the  data. 
Select  one  of  the  two  variables  (data  fields),  say  pilot,  and 
procede  to  rank  order  all  the  data  according  to  the  code  used 
(here  a numbered  coding  scheme  was  used;  the  analogous  procedure 
is  to  rank  order  pilots  alphabetically  by  their  last  name). 

Then,  for  each  pilot,  it  is  deslreable  but  not  essential  to  rank 
the  data  for  him  according  to  the  code  used  for  the  second 
variable,  here  it  was  symptom  (coded  with  alphabetic  characters). 
The  data  in  Table  19  have  been  ordered  by  symptom  within  pilot 
( le . ordered  first  by  pilot,  and  then  within  each  pilot  by  the 
symptom(s)  he  reported).  The  program  has  been  written  on  the 
assumption  that  the  data  will  be  in  this  prescribed  order. 
Computations  performed  will  only  be  correct  if  this  is  truly  the 
j case.  Care  must  be  taken  in  preparing  and  checking  the  input  to 

j ■ - assure  the  data  are  in  the  proper  order  prior  to  running  the 

j analysis,  or  the  computations  will  be  in  error  with  no  indication 

I *’  that  anything  has  gone  wrong. 


W 


TABLE  19 

DATA  SORTED  FOR  INPUT  TO  PHASE  I PROGRAM 


Pilot  No. 

Symptom 

61 

RHA 

62 

PAX 

62 

PAX 

69 

FAA 

69 

PAX 

69 

RHA 

69 

RHA 

72 

PAX 

72 

FAX 

72 

RHH 

74 

FAY 

74 

RHA 

74 

RHA 

74 

RHH 

74 

RHH 

76 

FAY 

76 

PGA 

76 

RHH 

91 

FAA 

91 

FAY 

91 

RHA 

91 

RHH 

91 

RHH 

The  Phase  I routine  simply  takes  a frequency  count  of  the 
unique  data  pairs  (specific  pilot-symptom  code  combinations)  that 
occur  in  the  data  deck  being  run.  This  frequency  count  information 
is  punched  out  on  cards  in  the  same  order  as  the  first  variable 
(pilot  here).  These  cards  (Table  20)  are  subsequently  used  as  the 
input  deck  for  the  Phase  II  routine.  The  output  of  that  analysis 
is  shown  in  Table  21, a.  Table  21, b provides  similar  output  for  the 
case  where  the  two  variables  are  reversed,  repeating  the  above 
procedures.  This  merely  entails  an  analogous  ranking,  but  this 
time  pilots  are  ranked  within  symptom. 

Given  the  output  from  these  two  Phase  II  passes,  it  is 
possible  to  compute  the  bivariate  uncertainty  (Us;P)  and  the 
bivariate  constraint  (Cs:p  or  Cp:s)  as  shown  in  section  3. 1.1. 2. 

The  conventional  matrix  format,  notation,  formulae,  and  compu- 
tations used  for  manual  calculation  of  Us,  Up,  and  Us:P  are 
given  in  Table  22. 
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TABLE  20 

DATA  COUNTED  (OUTPUT  OF  PHASE  I PROGRAM  RUN) 


Pilot  No. 

Symptom 

Count 

6l 

RHA 

1 

62 

FAX 

2 

69 

FAA 

1 

69 

PAX 

1 

69 

RHA 

2 

72 

PAX 

2 

72 

RHH 

1 

74 

PAY 

1 

74 

RHA 

2 

74 

RHH 

2 

76 

PAY 

1 

76 

PCA 

1 

76 

RHH 

1 

91 

PAA 

1 

91 

PAY 

1 

91 

RHA 

1 

91 

RHH 

2 

TABLE  21 


PHASE  II  OUTPUT 


a.)  Information  Calculated 

Pilot  No. 

Information 

61 

0 . 0000 

62 

0 . 0000 

69 

1 . 5000 

72 

0.9183 

74 

1.5219 

76 

1.5850 

91 

1.9219 

Uncertainty  in  Pilot  (Up):  2.6658 

Total  Uncertainty  (Up):  4.0018 

b.)  Information  Calculated 

Symptom 

Information 

PAA 

1.0000 

PAX 

1.5218 

PAY 

1.5850 

PC  A 

0 . 0000 

RHA 

1.9183 

RHH 

1.9183 

Uncertainty  in  Symptom  (U3):  2.3764 

Total  Uncertainty  (Up):  4.0018 

TABLE  22 


MATRIX  DERIVED  FROM  THE  RAW  DATA  SHOWING 
THE  JOINT  FREQUENCY  OF  PILOT-SYMPTOM  COMBINATIONS 

Since  p^j  = / N,  it  follows  that:  loggPj^j  = logg  / N) 

or  = logg  n^j  -logg  ^ 


Hence,  P^j 

log^ 

^IJ  = 

<"ij 

/ N)  logg 

-<"ij 

/ N)  log^  N 

Therefore, 

-I 

PlJ  ^’U 

II 

ij  / ^°g2  ^ij  +y  ^"ij  / 

ITJ 

i,j 

l.J 

Since,  y n.  . = 

N,  the 

above 

equation 

can 

be  more  conveniently 

1,J 

arranged : 

L 

L '’ij 

logg  P 

IJ  - 

N - 

VN^ 

"iJ  i°S2  "iJ 

S5rmptom: 

1-6 

Pilot 

FAA 

FAX 

FAY 

FCA 

RHA 

RHH 

I '"IJ  " 

1=1 

61 

0 

0 

0 

0 

1 

0 

1 

62 

0 

2 

0 

0 

0 

0 

2 

69 

1 

1 

0 

0 

2 

0 

4 

72 

0 

2 

0 

0 

0 

1 

3 

74 

0 

0 

1 

0 

2 

2 

5 

76 

0 

0 

X 

1 

0 

1 

3 

91 

1 

0 

1 

0 

1 

2 

5 

1=7 

y n,  , 

2 

5 

3 

1 

6 

6 

23  = 

L 

1=7  j=6 

1=1 

V 

I = “ 

1 1=1  j=l 

U = 


J=6  1=7  i=7 

log3  N [(Z-’lj) 

J=1  1=1 


n 


ij 


U = 


logg  N -1/^^ 


1=7 

z 

1=1 


1=6  1=6 

"ij) 


1= 


1=7  J=6 

Ugp  = N -1/N^  y 


Hij  logg  n^j 
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IDENTIFICATION 


///ro^MAT/OA/ 

Program  Title 


C4^t€tt7AtMTY  **>* *  Ci/iTs  Jf^PoItTirO 

Purpose 


/s-oaA 


y*/o/ 


Machine 


/6^ 

size 


pAftOS 


Tape/Cards 


Programmer 

Language 


s^3-o/-z 

File  Number 

S//6/U 

Date 


A 


SS  Zero 


INPUT 


1. 

Program  Object  Deck 

2. 

Header  Cards 

/\/0A/S 

Format 

. P* 

c 

~ 3. 

Data,  Serialized  

— 

^ Format 

(Z) 

. P- 

/ 

Y 

C 

4. 

Control  Card 

3zaa/k 

Format 

. P- 

L <; 
E 

5. 

Data,  Serialized  

— • 

^ Format 

<25 

. P- 

/ 

s 

6. 

Control  Card 

Format 

. P-  . 

7. 

End  Card 

A/of^e 

Format 

. P- 

SS 

0 B 

C D 

E F 

G 

OUTPUT 

Punch 




A>'*r»ie  C>>*Ctr'^*it¥y  0Cr«««  f~ 


Format 


Print  QtVft  /»'j^ . l/mm  ■T’mk/t  j y UiMt«r4*.iK4y  ptr  tyt/e  Attached  as  p, 


RESTRICTIONS 

1.  Data  Content:  See  Format  p (undesignated  - no  limits; 

A - alpha  only;  # - numeric  only;  0 - no  blanks;  X - no  symbols) 

2.  Sample  Size  Oo  rto't  use  mart  tZS"  /onssikit 

3.  Computational  Assumptions  ^r«  efu*f4y  />k*fy 

0*¥Se4  o/  fmi/ijrt 

4.  Computer  Assumptions 


ATTACHMENTS 

FORMAT  FORM/CODE  TRANSLATIONA-ISTING  OF  SOURCE  DECK/SAMPLE  OUTPUT 


* This  program  can  be  utilized  by  route  application  of  the  Instructions  on 

oage  one.  Additional  material  Is  provided  to  facilitate  modification. 
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AD-A054  447 


ARINC  RESEARCH  CORP  ANNAPOLIS  MO  F/6  15/5 

INFORMATION  TRANSMISSION  IN  OPERATOR  REPORTS  OF  EQUIPMENT  MALFU— ETC(U) 
MAY  67  H R LEUBA  AF  33(615)-3383 


X 


UNCLASSIFIED 


523-01-1-783 


NL 


sys«  IRiVi' 


suaj  |oMc  \^oo\  l^^oo^  ICriT 


12  3456789  101  I 2 3456789  I0|  I 23456789  I0|  I 23456789  10 


!%«.  ?.*U.  l-lUN.T  W 


m MtfiMMR 


IWftCK 

|o«o«< 


/NPur 


IDENTIFICATION 


I 23456789  lOM  23456789  I0|  123456789  I0|  123456789  10 


12  3456789  10  I 2 3456789  10  I 23456789  I0|  I 2 3 4 5 6 7 8 9 I0| 


dONTHOL  I'  23456789  10  I 23456789  10  123456789  10  123456789  10 


IDENTIFICATION 


12  3456789  10  I 2 3456789  10  I 23456789  10  I 23456789  10 


I 23456789  10  I 23456789  10  123456789  10123456789  10 


IDENTIFICATION 


12  3456789  I0|  I 2 3456789  I0|  I 23456789  I0|  I 23456789  10 


I 23456789  10  I 23456789  10  123456789  10  123456789  10 


IDENTIFICATION 


12  3456789  101  I 2 3456789  I0|  I 2345  6789  Oil  23456789  10 


IDENTIFICATION 


I 23456789  101  I 23456789  10  12345678910 


89  60 


CODE  TRANSLATION 


Page  Format  Col\imn  Entry 


«-7 

8-32 

S1-5H 

55- 5T 
59  -*» 
fc-S-  fc* 


Interpretation 


I 0CNT«^ICATI0M 
SY*A9TOK 


UH»T  OVlt^NATtOM  (UM  CODtO  MOKAACft  t \ZS ) 


PA1UU4C  W«»AWT 


6 if  o*k*Y  owe ***»Ki^** \wv«ieeJ 

3 *f  ow'y  owiO*  iweelyoA 

2>  if  fVeO*  owift  wee  iwy«twe4 


MAwruntTtoN  V^tti^vcATlON 


R0W  CHART  F(|)R  UNIVARIATE  ANALYSIS 


HI 


91 


ii! 


PARA‘M'l9r  09PS  A ' - - - -|  r 

C 

C LEURA  NO.  2 5/15/66  523-01 

C 

DIMENSION  N0?125a  • • 

JOO  

OIIHO.O 

D2l»l./L0GF«2.n  . . 

KARD/»0 

50  no  105  Il»l,125 

105  N0%ln#0  ^ 

N#0 

5'i  PRlNf  700  ' “ ' ’ " ' 

700  FORMAT^lHl, 2X6HSYS.  « , 2 3X7HSYMPT0M , 1 2X2HU1 , 2X2HC 1 , 4X2HU2 t 4X2HU3 , // 

!//□  ‘ * 

KCTtfO 

106  READ  100,NTEST,PR1,PR2,PR3,NU1»L1.NU2,NU3 

100  FnRMAT?X^,_4^A9,A7j5_22XI3,Il,I3,lXI3  a i . 

’ IF«NTESTn7,7,8 

8 N#N&1 
KCT^KCTCl 
KARD^KARDEl 
IF?L1-3p  9,10,11 

9 Nn^NUlDONO^NUln£2 

"■  '''Nh?NU2n#N05KNU2n£'2  " ' . . 

Nn?NU3n(IINO?NU3ne2 

no  PRINT  701,NTEST,PR1 ,PR2,PR3,NU1,L1,NU2,NU3 

701  FORMAT^IIH  , 3X  I 3 , 3X A9 , A9 , A7 , 1 5 X I 3 , 3X  1 1 , 3X I 3 , 3X I 3 a 
IF?KCT-45a  106,106,51 

10  NO%NUln#NOJN(llnGJ^ 

Nn^NU2o#N65iNU2n'6  3 
GO  TO  no 

11  N0^NIJlo#N01NUln£6 
GO  TO  no 

7 H#0. 

PRINJ  205 

7’05  FnRMAT3:iHl,20HUNiT  CODE  COUNT  ,//n 

KCT#0 

DO  12  I#l,125  ‘ 


IF95Nn3;ianl2,12,13 
13  DUMMY#NO?In 

H#HCDUMMY»L0GF^DUMMYn*D2 

PRINT  706,  I,NO?In 
706  FORMAT  * IH  ,4X13,9X15  D _ 
KCTOkCTCl 

IF  % KCT-45  n 12,12,901 
901  KCTIHO 

PRINT  705 

12  CONTINUE 
0UMMYiN*6 

HI*L0GFtDUMMYn*02-l./DUMMY*H 

JdfJCl  _ __ 

PRINT  702,N,’h,J 

702  FORMATS IHO. 13HCARD  COUNT  - ,I5.15H 


INFORMATION  .|F_10.4,_15H  SYMPT 
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I . 

C 


i 


lOM  NO.  ,13, //a 
DUDICH 

IF^SENSE  SWITCH  On  52,50 
52  D?»J 

Dll»01/D2 

PR  I N T 70<t,K&R0,Dl,  J 

70V  FORMAT* IHl, 13HCARD  COUNT^ 
lOM  NO.  ,I3,//a 
STOP  999 
END 


- ,15, iSH  INFORMATION  ,Fl6.A,15H  SYMPT 


PROGRAM  UTILIZATION  FORM* 


r 

;1 

FORMAT  SHEET 

SHEET  f OF  / 

svs«|«rpif]?|  AOJ  I 6U«J  lot»c  1 mod  I |moo2  |m»sc 


12  3456789  lOl  I 2 3456789  10  I 23456789  tO|  I 23456789  10 


10  20  30  40 


V/4«l( 


UNIT 

ISI 

UNIT 

IBI 

UNIT 

•4N««M 


/NPU  T ' 23456789  10  I 23456789  10  12345678910  123456789  I0| 


IDENTIFICATION  5 


12  3456789  10  I 2 3456789  10  I 2345678901  23456789  10 

20  30  40 


CoNTAOi- 


IDENTIFICATION 


IDENTIFICATION 
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CODE  TRANSLATION 


Page 

Format 

Colvunn 

Entry 

Interpretation  . 

/ 

(t> 

/~7 
8-32 
33-SV 
SX-SM 
SS-  64 

67 

*0*HTir»ftAT*OM 

SYM^TOIA 

IAAiht«N AMe«  (m:»$i  0tf  «/•<#•  ^vncAtfW  *<  m/m«  / ■— 

NumMA  oa  maiht.  i«x«N 

UNITS 

KAbTUNCtlOM  V««ITt  CATION 

1 

PROCEDURES  FOR  USING  THE  ANALYSIS 
OF  VARIANCE  PROGRAM 


1.  Prepare  Input  data  in  the  same  form  as  for  the  Univariate 
Analysis  (same  deck  can  be  used). 

2.  Stack  the  input  deck  behind  the  program  deck  prepared 
according  to  the  following  program  listing. 

OUTPUT; 

a)  a list  of  the  input  data 

b)  the  mean  and  standard  deviation  for  each  symptom  for  each 
variable* 

c ) a summary  ANOVA  table  for  each  variable 

NOTE; 

a)  the  standard  deviation  is  a biased  estimate,  uncorrected  for 
degrees  of  freedom;  this  permits  a simplification  in  the 
program  by  assuring  that  a zero  denominator  does  not  result 
in  those  cases  where  the  "variable"  was  only  one  score,  since 
in  that  case  N-1=0.  If  an  unbiased  estimate  were  required, 
the  program  must  be  appropriately  modified. 

b)  negative  data  in  the  output  listing  simply  indicates  the 
value  was  missing,  and  the  point  is  not  counted. 

*c)  variables  analyzed  Include;  preparation  time,  malfunction 
verification  time,  fault  location  time,  part  procurement 
time,  repair  time,  administrative  delay  time,  number  of 
three  level  technicians,  number  of  five  level  technicians, 
and  number  of  seven  level  technicians. 


a o u o 


11 
I t 

o 

0 

1 

r 

6 

5 


ARAMI9I  09PS  A 

LEUBA  #3  ” ’ . ■ 

DIMENSION  CSUM19a,CX2?9n,T6TAH9a,TSS*9D,X?9D,SIGX9n,CSSt9n,CN0*9o 

l,TLDF«9n,Ayi9a_ 

DO  15  I#l,9 
TL0F*In#^.0  _ 

TOTAiltlnUO.O 

TSS*Ia#0.0 

15  CSS*In#0.0 

^F/HO^O 

10  DO  16  I#l,9 
CN0tIn(»0.0_ 

CSUM^IndlO.O 

16  CX2*In(»0.0 

1 KOUNTUb  “ 

_ PRINT  200 

2 00  FnRMAT?lHl,2X5HIDENT,12X7HSYMPT0M,  13X4HPREP',  3X4HMIv.,’3X4HF.L.,3X4H 
IP.P.  , 3X4HREP._f  3X^M.  AD.  i 3X4HN0.  3 , 3XAHN0. 5 , 3X4HN0. 7 , ///□ 

5 READ  I00,IT,01,D2tD3,D4,*X*In,l#l,9a 
100  FORMAT*  13, A4,2A8,A9,5F3.2,F4.2,3F1.0  n 
IFflTn  50,50,17 

17  PRINT  2Ql,IT,01,D2,D3,D4,fX*!n,lil,9a 

201  FORMAT^IH  , 1 X I 3 , A4 , 2 X2  A8 , A9 , 1 X5  t3XF4. 2a , 3XF  5 . 2 ,' 3J3XF4  . inn  ' 
KOUNTIKKOUNTGI 

DO  18  I mi, 9 
IF*X*Ina  18,190,191 
191  CSUM*In#CSUM*Ia£X*Io' 

CX2*lail>CX2*la£X*la»X*ln 

■ 190  CNO*In#CNOf intl.  - - — 

18  CONTINUE 
IF*K0UNT-45a  5,1,1 

50  CDF/KCDFCl.  _ 

DO  19  I « 1,9 

TLDF*lnl>TLDF*la£CNO*Ia _ _ 

TnTAL*In#TOTAL?In£;CSUM?In 

TSS*IajTSS*InGCX2*In 

AV*In(»CSUM*Ia/CNO%In 

SIG*Iaf SQRTF**CX2*In-*CSUM*In*CSUM*In/CN0*InnD/CN0*Inn 

19  CSS*ia#CSS*InCCSUM*ln*CSUM*in/CNO«In 

PRINT  202,lAV*ln. 101 ,90 

202  FORMATTING, 25X7HAVERAGE,7X5*F4, 2, 3Xn,F5. 2, 3*2XF5.2aa 
PRINT  203,?SIGTlo, Iil,9n 

203  FORMATTIH  , 25X5HSIGMA,6X5T2XF5. 2 n,3XF5.2,  3T2XF5.2oa 
PRINT  607,TCN0TIn,I(»l,9a 

607  FORMATTIH  , 25X 1 HN, 1 2X5TF 5. 0 , 2Xo ,F6. 0, 3T2XF5 .Odd 

IFTSENSE  SWITCH  0 a 75,10 

75  DO  20  I (» 1, 9 
PRINT  204 

204  FORMATTIHI , 10X26HAN0VA  WITH  UNEQUAL  COLUMNS, ///a 
GO  TO  *21,22,23,24,25,26,27,28,290,1 

21  PRINT  301 

301  FORMATTIH  ,13X20H  PREPARATION  TIME  ,//n 
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Gn  TO  30 

22  PRINT  302 

302  FORMATTIH  ,13X20H  MALF.  VER.  TIME  ,//o 
GO  TO  30 

23  PRINT  303 

303  FORMAT%IH  ,13X20H  FAULT  LOG.  TIME  _ xJ ISL 

GO  TO  30  ' ' ' 

2A  PRINT  30V 

30A  FORMAT^Th  ,13X20H  PART  PROC.  TIME  ,//n 
GO  TO  30 

25  PRINT  305 

305  FORMAT^IH  ,13X20H RJ.PA  IJV  J 1 _t//n 

‘go  Trj“30 

26  PRINT  306 

306  FnRMAT3:iH  ,13X20H  MALF.  ADMIN.  DELAY  ,//n 
GO  TO  30 

27  PRINT  307 

307  FORMAT^lJl  ,13X20H  NO.  OF  3 LEVEL  MEN  ,//a 
G0'~T0'30 

28  PRINT  308 

3C8  FPRMAT?1H  ,13X20H  NO.  OF  5 LEVEL  MEN  ,//n 
GO  TO  30 

29  PRINT  309 

309  FORMATTIIH  , 13X20H  NO.  0F_7  LE VE L_M_EN_  , / /n 

30>RfNT‘310~ 

310  FORMAT’JIH  , 10X26HANALYS  I S OF  VARIANCE  T ABL  E , /// 3X6HS0URCE , 8X2HSS  , 
17X2HDF, 7X2HMS,9XlHFn 

CF#T0TAL2Ia*T0TAL?Ia/TL0F?In 

A«CSS'SIn-CF 

BACDF-L. . 

C«A‘/ft 

H/fTSSTII  a-CF 
AIOTLDF-I. 

E«H-A 
F«A 1-8 

G«E.^ 

6«<C/G 

PRINT  311,A,B,C,D,E,F,G,H,AI 

311  FnRMATT:iH0,2X6HC0LUMN,3XFll  .5, 1XF5.0,2XF10.5,2XF6.2/3X6HERR0R  , 3XF 
111.5,1XF5.0,2XF10.5/3X6HTOTAL  , 3XF 1 1 . 5 , I XF5 . Od 

20  CONTINUE 
STOP  777 

END* 
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PROGRAM  UTILIZATION  PORM» 


,1 


FORMAT  GHx:.£T 


SHEET . 


1. OF 


VA«  A IvAA  b 


12  3456789  10  I 2 3456789  lOj  I 23456789101  23456789  10 


20  3 


H^aobr 


IDENTIFICATION 


1 23456789  10 

1 23456789  10 

12345678910 

123456789  10 

50  6 


Nq  TfC^S 


12  3456789  10 

1 2 3456789  10 

1 2345678910 

1 

0 2 

0 3 

IDENTIFICATION 


I 23456789  10  I 23456789  10  I23456789I0I23456789  10 


50  60  70  80 

ATEST  1 BTEST  \ 


123456789  10  123456789  10  I 2345678910123456789  10 

20  30  40 


UTPV7 


IDENTIFICATION 


I 23456789  10  I 23456789  10  I23456789I0I23456789  10 

'T  


50  60  70  80 

COUNT  \ 

'117  I IT  @ 

I2  3456789  10112  3456789  I0|  I 23456789  I0|  I 23456789  10 


c 


IDENTIFICATION 


12  3456789  10  12  3456789  10  I 23456789101  23456789  10 

20  30  40 


IDENTIFICATION 


I 23456789  10  I 2345678  9 lOj  123456789  I0|  123456789  10 


50  101  60  70  8( 


CODE  TRANSLATION 


Page  Format  Column  Entry 


aa 


Interpretation . 

2 “3  V4«4  f*r4,o  (A’oamatQ))  a 

if- s V4AA  C 104,0  /evyf  \/AA/0a*A  ^ 

10- ft  VAAB  SrAATtM*  fAfoco  AOA  ^AAtABf.0 B 
/2-f3  VAAB  ^*04.0  000  VAAtABlo  <3 


8-/2 

n~n 

23-26 
/ -S' 


Do  Not  ^tCBoo  A'tt^e  saacos  000  i/AA/AO^^ 

Punch:  VAAtAQi^  /!  fro00*Ar(^ 

PuNCH>  \/AAfAa*A  a f^OA/AAf® 

/OUNCH;  A/o  *>0  Casas  'n  tecc  foeffrtA/A^o 

No  00  ''f  NsTAfU  (A^tfT^  CAtii  Af0T  Cm^NTAo) 
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OlOlO 

JOB 

LEUBA  518-01  7/8/66 

01020 

CTL 

6611 

010  iOBEGIN 

sw 

1.5 

01040 

sw 

9,13 

01050 

R 

01060 

MCW 

3,SET£3 

01070 

MCU 

ll,SETe6 

01080 

MCW 

7,C0R1C3 

01090 

MCW 

7,CnR3Co 

01100 

MCW 

15,C0R2e3 

OHIO 

MCW 

15,C0R4e3 

01120 

cs 

80 

01130SET 

sw 

0,0 

01140 

R 

01 150RET 

LCA 

aoooa. iRi 

01160C0R1 

LCA 

O.TESTl 

01170C0R2 

LCA 

6,TEST2 

01 180CnR5 

C 

O.TESTl 

01190 

BU 

PUNCHO 

01200C0R4 

C 

0,TEST2 

01210 

BU 

PUNCHO 

01220 

MA 

aooia. iRi 

01230 

R 

01240 

B 

C0R3 

01250PUNCH0 

MCW 

TESTl. 112 

02010 

MCW 

TEST2, 119 

02020 

MCW 

IRI, 126 

02030 

P 

02040 

BLC 

ENDJOB  LAST  CARD  MUST  BE  BLANK 

02050 

B 

RET 

02060ENrJ08 

CC 

1 

02070 

LCA 

aEND  OF  JOB  a,  215 

02080 

W 

02090 

CC 

1 

02100 

H 

02110TEST1 

DCW 

#50 

02120TEST2 

DCW 

#50 

02130IR1 

EQU 

89 

02140 

END 

BEGIN 

PROGRAM  UTILIZATION  FORM* 


IDENTIFICATION 


3/ VAR/ ATS  A/VAL 

YS/S  Jl 

s/e-o/-z 

Cm* *  SyCMt 
C4t.t**t-AT0  Row  (J  / leTAt.  (J 

Program  Title 

/ S-OSA 

Pile  Number 

a/7/u 

Purpose 

Programmer 

Date 

04/eos 

SbffrRAAf  JT 

y^s 

Machine  Size 

Tape/Cards 

Language 

SS  Zero 

INPUT  Output  V S/8’Of^/ 

1.  Program  Object  Deck 

/ •*/« 

r I/44.0SS 

(D 

2.  Header  Cards 

A/oa/S 

Format 

p. 

OUTPUT  Sv*l  y — tfMtM  tftH/ 

Pvmch  Totm*.  Som  / Tor^t. 

Print  SAM£^ 

RESTRICTIONS 

1.  Data  Content:  See  Format  p (undesignated  - no  limits; 

A - alpha  only;  # - numeric  only;  0 - no  blanks;  X - no  symbols) 

2.  Sample  Size 

3.  Computational  Assumptions  /Pea^ /Vt  Loj  A n </oC 

4.  Computer  Assumptions  x 

ATTACHMENTS 


0) 

Format 

Attached  as  p.  


FORMAT  FORM/CODE  TRANSLATIONA-ISTING  OF  SOURCE  DECK/SAMPLE  OUTPUT 


* This  program  can  be  utilized  by  route  application  of  the  Instructions  on 

page  one.  Additional  material  Is  provided  to  facilitate  modification. 
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nio  noon 


PARAMI9I0808PS  A " ^ " 

PROGRAM  TO  COMPUTE  JOINT  UNCERTAINTY  — PHASE  II 
LEUBA,  WO  518-01*  JUNE  7*  1966 
INPUT  IS  OUTPUT  OF  PHASE  I 
NO  HEADER 

_ _NP  S_SW,  NO  TAPE,  PRINT  £ PUNCH  ON 

END  DATA  WITH  A BLANK  CARO 
NROW  #0 
RSUM  #0. 

RINFO  n 0. 

TSUM  M 0. 

TINFO  # 0. 

VINFO  #0.  ■ 

1 FORMAT  ?7XA5, lOXFA.O  n 

2 FORMAT?7HROW  D ,A5,3X7HROW  NO  ,I4,2X8HROW  SUM  ,F5.0, 2X6HINF0  , 

lF9.5a  _ 

3 FORMATflH  , 3X A5 , 4X I 4 , 2XF6. 0 , 2XF9, 5a 

4 FORMAT  %1H-,2X, 17HINF0  IN  MARG I NAL S , 6X , F9. 5a 

5 FORMAT  'f  1H-,2X  , 17HT0TAL  I NFORM  AT  I ON,  6X  , F9.  5'a 
8 FORMAT  %F9.0a 

6 FORMAT  «1H1,3X,29HR0W  D ROW  NO  ROW  SUM  INFOa 

7 FORMAT  %I3a 
PRINT  6 

D2  M l./L0GF72.n 
READ  1,ATEST,DATA 
70  RSUM  # RSUM  £ DATA 

RINFO  H RINFO  £ DAT A*LOGF«DATAa 
DUMMY  M ATEST 
30  RFAD  1, ATEST, DATA 
D0MMY#ATEST 


APPENDIX  VI 


LIST  OP  CALCULATIONS  PERFORMED 

A.  Univariate  Uncertainty  Calculations  - ASB-4 

1.  Intermittent  vs.  stable  symptoms 

2.  No.  of  Phrases 

3.  Malfunction  number  for  1 card  events 

4.  Malfunction  number  for  multicard  events 

5.  2-dlglt  subject 

6.  2-dlglt  description 

7 . Mood 

8.  Modifiers 

9.  One  card  events 

10.  Number  of  cards  per  event 

11.  Operator  complaints  without  modifiers 

12.  Operator  complaints  by  2-dlglt  modifiers 

13.  Subject  by  description 

14.  Carefully  sorted  Job 

15.  Job  as  2-dlglt  subject 

B.  Analyses  of  Variance  - ASB/4 

1.  Intermittent  vs.  stable  symptoms 

2.  Preventive  maintenance  by  subject 

3 . Mood 

4.  Malfunction  number  by  number  of  cards /events 

5.  2-dlglt  description 

6.  Job  as  2-dlglt  subject 

7.  Carefully  sorted  Job 

8.  Job  by  maintenance  men  profile 
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C . Bivariate  Uncertainty  Calculations  - ASB/4 

1.  Subject  by  modifier 

2.  4-dlglt  description  by  second  modifier 

3.  Subject  by  adjective 

4.  Subject  by  maintenance  men  profile 

5.  Subject  by  malfunction  number 

6.  Description  by  adjective 

7.  Maintenance  men  profile  by  malfunction  verification 

8.  Subject  by  description 

D.  Bivariate  Information  Calculations  P-106 


1. 

Pilot  vs.  A/C  No. 

1965  SQD  95  (Dover) 

2. 

Pilot  vs  symptom 

3. 

Maintenance  man  vs 

symptom  (Dover  only) 

4. 

A/C  vs  symptom 

(Dover  only) 

5. 

Pilot  vs  A/C 

6. 

Symptom  vs  NFS 

7. 

SQD  vs  Symptom 

8. 

S3miptom  vs  unit 

9. 

Verlf  vs  symptom 

10. 

(Method  of  TS  & Malnt.  Concept)  vs  symptom 

11. 

(Malnt  Process  (3  col  field)  vs  NFS 

12. 

Symptom  vs  Downtime 

13. 

Date  vs  symptom 

14. 

Verlf  vs  NFS 

15. 

Pilot  vs  verlf. 

16. 

NFS  vs  pilot 

17. 

How  Mai  vs  Man 

18. 

Unit  vs  How  Mai 

19. 

NFS  vs  Unit 

20. 

How  Mai  vs  NFS 

21. 

A/C  vs  symptom 

(Selfrldge  only) 

22. 

Pilot  vs  symptom 

(Selfrldge  only) 
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Malnt.  concept  vs  T.S.  method  (Self ridge  only) 
Maint.  concept  vs.  T.S.  Meth.  (Self ridge  only) 
A/C  vs  date 
Pilot  vs  date 
Man  vs . verlf . 

(Malnt.  concept  & T.S.  meth.) 

Verlf.  vs  (malnt.  concept  & T.S.  meth.) 

Pilot  & symptom  vs.  Verlf. 

Pilot  & symptom  vs  A/C 
A/C  vs  verlf. 

Base  vs  verlf. 

Mission  success  vs  verlf. 

Malf.  no.  vs  verlf. 

Severity  vs.  verlf. 

Action  taken  vs.  verlf. 

Concept  vs.  verlf. 

T.S.  vs.  verlf. 

Downtime  vs.  verlf. 

NFS  vs  downtime 
NFS  vs  malnt.  man 
Malnt.  man  vs  T.S.  Meth. 

T.S.  Meth  vs  (symptom/Unlt ) 

(Symptom/unit)  vs  NFS 
NFS  vs  Malnt.  concept 
Malnt  concept  vs  symptom 
Ssmiptom  vs  base 
Symptom  vs  (base  and  AC) 

(Base  & AC)  vs  NFS 
NFS  vs  T.S.  Meth. 

T.S.  meth  vs  How  Mai 
How  Mai  vs  symptom 
Report  no.  vs  symptom 
How  Mai  vs  symptom 
Pilot  vs  How  Mai 


110 


4K 


57.  Downtime  vs  pilot 

58.  Unit  vs  Downtime 

59.  When  discovered  vs  unit 

60.  Downtime  vs  when  discovered 

61.  Downtime  vs  meth.  of  T.S. 

62.  Meth  of  T.S.  vs  Symptom 

63.  Symptom  vs  Malnt.  concept 

64.  Downtime  vs  malnt.  concept 

65.  Unit  vs  pilot 

66.  A/C  vs  Unit 

67.  Mission  success  vs  severity 

68.  Man  vs  NFS  (Dover  only) 

69.  Meth.  of  T.S.  vs  man  " " 

70.  Downtime  vs  meth  of  T.S.  " " 

71.  Man  vs  Downtime  " " 

72.  Man  2 vs  Man  1 " " 

73.  Downtime  vs  Man  2 " " 

74.  Unit  vs  downtime  " " 

75.  Shop  vs  Unit  " " 

76.  Symptom  vs  Shop  " " 

77.  Downtime  vs  symptom  " " 

78.  How  mal  vs  downtime  " *' 

79.  Shop  vs  How  Mal  " " 

80.  Man  vs  shop  " " 

81.  Method  of  T.S.  vs  man  " " 

82.  Shop  vs  method  of  T.S.  " " 

83.  Malnt.  concept  vs  shop  " " 

84.  Downtime  vs  malnt.  concept  " " 

85.  Pilot  vs  downtime  " " 

86.  (Symptom  & Unit)  vs  pilot  " " 

87.  Downtime  vs  (Symptom  & unit)  " " 

88.  (Symptom  & Man)  vs  downtime  " " 

89.  Concept  vs  man  " " 

90.  How  Mal  vs  Malnt.  Concept  " *' 


91.  Pilot  V3  symptom  (Dover  only) 

92.  A/C  vs  (pilot  vs  symptom)  " " 

93.  A/C  & Pilot  vs  symptom  " " 

94.  Unit  & How  Mai  vs  man  " " 

95.  Concept  vs  meth  of  T.S. 

96.  NFS  vs  (Meth  & concept) 

97.  Symptom  vs  method  of  T.S. 

98.  A/C  vs  Symptom 

99.  Base  vs  A/C 

100.  Pilot  vs  NFS 

101.  Pilot  & symptom  vs  NFS 

102.  Pilot  & symptom  & unit  vs  NFS 

103.  Unit  vs  How  Mai 

104.  Unit  & meth.  of  T.S.  vs  How  Mai 

105.  Unit  & Meth.  of  T.S.  & pilot  vs  How  Mai 

106.  Symptom  vs  downtime 

107.  Symptom  & Unit  vs  downtime 

108.  Symptom  & Unit  & Meth  of  T.S.  vs  Downtime 

109.  Symptom  & Unit  & Method  of  T.S.  & Maint.  concept 

vs  Downtime 

110.  Symptom  vs  unit 

111.  Symptom  & Pilot  vs  Unit 

112.  Symptom  & pilot  & Meth  of  T.S.  vs  unit 

113.  Symptom  vs  verlf. 

114.  Pilot  & s3nTiptom  vs  Verlf. 

115.  Pilot  & sj^ptom  & maint.  concept)  vs  verlf. 

116.  Unit  vs  bench  check  (FIHH  Ssnnptoms  only) 

p 

2.  X analysis  of  P-IO6  shop  symptoms 

a.  No.  words 

b.  No.  phrases 

c.  Presence  of  word  denoting  intermittent 

d.  Presence  of  qualifying  phrase 
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e.  Number  of  nomlnallzatlons 

f.  Presence  of  negative 

g.  Position  and  presence  of  subject 

h.  Position  and  presence  of  object 

I.  Position  and  presence  of  verb 

J.  Position  and  presence  of  prepositional  phrase 
k.  Position  and  presence  of  modifiers 

3.  Correlation  analysis  for  selected  ASBA  symptom 

a.  Information  content  versus  trouble-shooting  time. 

b.  Information  content  versus  probability  of  malfunction 

verification 

4.  Correlation  analysis  for  selected  F-106  symptoms 

a.  Information  content  versus  probability  of  malfunction 

verification 

b.  Information  content  versus  number  of  units  removed 

per  action. 


1 

I 
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The  ASB-4  data  (flight -line  reports)  and  the  shop  data  for  the  MA-1 
contained  written  descriptions  of  the  malfunction  symptoms.  To  make 
these  data  amenable  to  multivariate  uncertainty  analysis,  a coding 
scheme  designed  to  retain  the  grammatical  form  and  content  of  these 
symptom  reports  was  used. 
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